Reply to post: Re: Still not understanding

Visa fingers 'very rare' data centre switch glitch for payment meltdown

SImon Hobson Silver badge

Re: Still not understanding

Why it took so long to disable the failing switch once it was identified

As already said, the guys that wold have been able to diagnose this AND do something about it have all gone. The people running it now will probably be junior techs on a different continent with a) manglement imposed limits on authority and b) culture imposed limits.

The latter is important. For many of us in northern Europe it's seen as a good trait to be able to sit down, look at the evidence, and formulate a theory as to what is wrong - and formulate a plan for how to fix it. So as already said further up the comments, a good ops team would probably have had it fixed before many people realised there was a problem.

But AIUI, in many of the places such functions are offshored to, there is a different culture - where individualism is frowned upon, and the techs are supposed to "just follow the flowcharts". In such a culture, to get the offending switch powered off would require the problem passing up many manglement levels, endless meetings, and above all - discussion of who takes the blame.

A secondary factor is the modern disease of not supporting people to make decisions. So even if a techie did realise that "all it needs is to power cycle this switch" - it's a very secure person who can take on that decision and expect his manglement chain to support him in doing so. More normally, the "safe" option is to do nothing - it's not your fault the system failed. But go and do something that should fix it, but for some reason doesn't - well your head is on the block for doing it.

Go and read some of the "the day I ..." stories in ElReg - and in particular the comments. Some of the best ones involve the person "doing something" but being supported by their managers on the basis that "the only person who never made a mistake was the one who never did anything".

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019