Re: Learn from this and sympathise
Let's be honest. I know that here we have not done any sort of backup test for ages. I'm willing to bet that most (if not all of you and the register too) haven't either. Businesses arn't normally all that keen to let you take systems down to do this.
Oh no, we test. There's a simple reason for that: we also have BCM competence and we class a failing backup as a company extinction event - if we cannot recover IT within 48h we hit all sorts of problems, including regulatory ones. This makes it easy to defend such tests, and we have two data centres that each are capable of taking a full load for a few hours (at impaired performance, but at least it doesn't all come to a grinding halt) so we basically kill one off completely, every 6 months (it means that in a year we have both tested).
That said, it is NOT a very comfortable thing to do. Even though we know it all works in theory, hearing it all spin down at once is nerve wrecking (first we kill power to see if UPS + generators pick up, but after that we kill main power to simulate a catastrophic event). I think we may put the next one on video for our customers, that gives us at least some marketing capital as a return on effort..
Biting the hand that feeds IT
