Reply to post:

How an over-zealous yank took down the trading floor of a US bank

fixit_f

I worked for a large investment bank in the late 2000's with multiple trading desks and about 2000 employees, plus two huge server farms spread across two datacentres for DR purposes. The whole trading infrastructure was an absolute rat's nest of duplicated systems and people's vanity projects which all had to be kept in sync with a sprawl of horrible middleware and half-arsed trade feeds, all built completely differently and inconsistently. One day one of the tech team managed to issue a remote shutdown with some kind of crazy wildcard that rebooted every Windows desktop and server across the whole organisation. The time it takes the machines to come back up is trivial, but in a normal (for example post patching) environment the machines would normally all be brought up in tiers in a particular order - domain controllers first, then things like email servers, then servers housing SQL Server databases, then servers housing application engines, then finally things like desktops. This was important as a lot of the services were set to auto-start and they'd get confused if things they depended on weren't already up. Once all these machines had been brought up the application teams would start bringing up remaining services, synchronising FIX sequence numbers for trade feeds, resolving issues with things that are partially processed or transactions rolled back as part of the shut down etc etc. So resumption of trading varied desk by desk depending on their systems, but in many cases they weren't able to trade for several hours and ironing out every single issue and making sure all trade and pricing data across all systems was correct and synched up took days. On the desk I worked for, what was the impact of not being able to trade? Well - after hours of the traders screaming at us that they had positions to close out, markets had moved unexpectedly and the P+L on the positions they'd been unable to close out was actually substantially up. That one IT guy literally made millions for the bank - he still works there now in fact.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019