How an Amazon engineer's slip-up started a 20-hour Netflix cock-up An Amazon engineer hit the wrong button on Christmas Eve, deleting critical data in its load balancers and ultimately knackering vid streaming biz Netflix for 20 hours. The Netflix outage hit customers in the US, Canada and Latin America on 24 December, particularly those using games consoles and mobiles to watch films, while …
I'm getting rather fed up of the argument from some that this is all down to the change processes and that heads should not roll. I know my organisation's efficiency would improve immensely if I were allowed to fire some asses now and then rather than just shuffle them off to the side to some role where (I hope) they cannot do any damage. I often wish that management had not downsized HR quite so much so that there were actually some warm bodies who would help me satisfy all the regs for sacking someone so I could use the money to hire someone decent instead...
No wonder you were AC on that comment. You think it should be ok to sack people because managers like you continue to ask them to do things that they aren't trained for, don't have the time to finish, isn't their responsibility and that you already outsourced or downsized the team that was SUPPOSED to do that job. Oh, and on top of that you give them a 2.5% pay 'increase' then blame the market conditions.
To err is human, to really fuck things up requires a computer, a tired engineer and piss poor management.
> there were actually some warm bodies who would help me satisfy all the regs for sacking someone
Wow definitely not an American in a right to work state then. Right to work someplace else no questions asked is what it should be called. It sounds worse than it is though in that it is generally easier to find a job as their is less risk in hiring someone but you are lucky if you find a place that treats you as anything but an asset though.
Actually I'm the kind of manager that fights tooth and nail to get my team trained, proper pay rises, promotions and fight against outsourcing and downsizing. I have hated it in the past when I have had to make good people redundant. I don't ask any of my team to do anything that I cannot do myself. All of which is why I'll never rise any further. However the propensity of some people to take the piss does make life worse for everyone else. If you know your UK employment law and your employer stints on HR you can be almost unsackable.
And 2.5%. I'd love to be able to secure that kind of rise for the best people in my team.
" I don't ask any of my team to do anything that I cannot do myself"
You are either the most talented person in the world, run the least skilled IT department in the world, or the best bullshitter in the world, or as you don't seem to expect your staff to do stuff you can't do, could explain your own lack of promotion.
This post has been deleted by its author
Would it not make sense to have some proper change control and then Amazon could have reviewed their change documentation and hey presto notice a change had happened around the time of the problem. Also while they are at it wouldn't it also be a good idea to have a change freeze around such a critical time of year! Unbelievable
So the problem is...
(a) Netflix have poor business continuity planning and rely on a single supplier (AWS) for its systems.
Cause: Poor management decisions/understanding
(b) AWS have poor processes that allow a single point of failure
Cause: Poor management decisions/understanding
(c) Netflix believed the "cloud" of Amazon would be redundant against anything and assumed they had covered the issues in (a)
Cause: Poor management decisions/understanding
The real issue isn't the engineer that "made an error" but the AWS system that can fail despite supposedly being uber-geo-redundant and so on, and the Netflix management who decided to put the eggs in one basket.
As I understand it, Netflix have local content caches with various ISPs so I assume the issue was the database/account side and not the underlying content availability - so it would be a *relatively* less expensive task to put a better system in place (I'm not pretending it is trivial, but it's obviously "less tricky" when you haven't got to replicate what I assume is a huge amount of content which would be costly to store/stream en masse
A better fix would have been to have multiple providers and the ability to have the Netflix client(s)/website(s) detect/choose/forced etc.
Of course this would require more expenditure and at £5.99 (or it seems a penny more if you subscribed more recently) it's unlikely there's enough margin I guess.
It really says something about the Amazon change control process. It also says volumes about their support staff; both the person that did the deleting and the subsequent ones that did the troubleshooting. When they encountered missing data, the first thing should have been to look at who made a change, what the change was and what was actually changed. I think Amazon needs to invest in an AAA solution.
... it should not be possible for a single engineer to wreak this kind of havoc: systems like this should be resistant even to deliberate malice. Your engineer could be tired, inexperienced or unwell. But they could also be a saboteur working for a competitor, an employee with a grudge, a criminal who is going to hold your system to ransom or even an out-and-out terrorist.
Now all we need if for the engineer, PFY or intern who went on xmas vac forgeting to flick the switch to update the TV Guide data in Media Center to sort out the updates there (we know BDS Ltd have sent data packages to MSFT for upload) then everyone will be happy :-) (For ref UK data ended Jan 1 so having to use dead tree TV guides and ending up with loads of "Manual Recordings" :-( )
the idiot who didn't check his/her work should shoulder some responsibility for this. perhaps, before initiating a change that could cause a major service outage during a peak usage period, they should take a minute to really look at what they've instructed the system to do before they hit the go button.
i can't see how this is management's fault: it's just poor workmanship.
i'm assuming it's all techies who have blamed the managers. well, i am a techie and this is just someone doing a shit job because it's xmas eve and they're not paying attention.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
