Re: Still not understanding
Why it took so long to disable the failing switch once it was identified
As already said, the guys that wold have been able to diagnose this AND do something about it have all gone. The people running it now will probably be junior techs on a different continent with a) manglement imposed limits on authority and b) culture imposed limits.
The latter is important. For many of us in northern Europe it's seen as a good trait to be able to sit down, look at the evidence, and formulate a theory as to what is wrong - and formulate a plan for how to fix it. So as already said further up the comments, a good ops team would probably have had it fixed before many people realised there was a problem.
But AIUI, in many of the places such functions are offshored to, there is a different culture - where individualism is frowned upon, and the techs are supposed to "just follow the flowcharts". In such a culture, to get the offending switch powered off would require the problem passing up many manglement levels, endless meetings, and above all - discussion of who takes the blame.
A secondary factor is the modern disease of not supporting people to make decisions. So even if a techie did realise that "all it needs is to power cycle this switch" - it's a very secure person who can take on that decision and expect his manglement chain to support him in doing so. More normally, the "safe" option is to do nothing - it's not your fault the system failed. But go and do something that should fix it, but for some reason doesn't - well your head is on the block for doing it.
Go and read some of the "the day I ..." stories in ElReg - and in particular the comments. Some of the best ones involve the person "doing something" but being supported by their managers on the basis that "the only person who never made a mistake was the one who never did anything".