Fire, fire! Just move your data centre onto my lawn ... Oh rats! Welcome again to On-Call, our regular weekend feature in which readers share the odd things they've been asked to do at odder times of the day. This week's contributor has asked to remain anonymous, because his tale of a curious incident concerns a company that provided services to the emergency folk, among others. Our reader …
Exactly why anything mission critical needs multiple elements of resilience.
When your primary service fails and you kick into Business Continuity, your BC is now your Primary, so what's your new Businss Continuity plan?
The vast majority of businesses don't need to stand up that third data centre. But at the very least the plan must be written down and tested.
This post has been deleted by its author
All BC is a cost/benefit analysis. What are the odds of your primary DC failing, and how much does it cost you if it happens? That can inform the decision of how much you pay for implementing your BC strategy, and whether it involves a second DC, a hot site, or whatever. The odds of both your primary and secondary DCs failing ought to be really low (or you need a new provider and/or better geographical separation) so you really don't need to plan for this unless even those odds when measured against the monetary loss justify real planning for "what if both DCs are taken out". If you're a Fortune 100 company, sure, but not for some local service provider like in the story.
The real problem with the place in the story was that it took FOUR DAYS to repair a cut fiber IN THEIR PARKING LOT (so no need to work with the city to block off streets etc.) Who the hell was their telco, I want to know so I never have anything to do with them!
"All BC is a cost/benefit analysis. "
It is, and that is some a lot of people forget. How likely is a disruption? How likely is it that your primary systems will be disrupted at the same time as your backup systems? How much will any disruption cost? How will any disruption impact the business? How much will maintaining (potentially multiple) backup systems cost?
It reminds me of an old Jasper Carrot joke about a shop using £60,000 of CCTV and security equipment to protect £25 worth of Smarties.
The solution is easier if you are a massive international corporation as they are likely to know exactly how much they will lose due to primary systems going down, and it is likely to be considerably more than maintaining backup systems that may not be used 95% of the time.
I was talking to someone who manages security at a major bank (won't say which one, but they are large world wide, just not one of the big four here). In the last few years, they moved out of their old UK headquarters in to a new one. They are still maintaining their old UK headquarters though (even though the building is not used), purely as a backup so the bank can carry on trading in the event something happens that takes out their new HQ.
Backhoe in the car park doesn't necessarily mean anything as tidy as a cut fibre, though. Our city-centre office went dark one day, and a look out the window revealed a JCB, a hole in the yard, and a lot of people peering into the hole. Unfortunately the digger hadn't so much cut the main power cable as pulled it, hard. 10m away was the service chamber, where all the switchgear and fuse panels had been ripped clean off the wall, and all the internal wiring to them torn off.
Repairing a cable like that is easy, but I was serioulsy impressed by just how fast the contractors got the distribution panels rebuilt and rewired. Being a development group with no particular deadline at that moment our BC plan was to retire to the pub across the road for the afternoon. Like any well-tested plan, it worked like a charm.
I was working for a company (I won't mention who) and we were moving services to a new DC in Ireland. We had two fast redundant links - one going via Wales and the other going out through Scotland.
However.....
One night, the UK experienced very high winds - which blew a wall down destroying the Welsh link. At the same time, a ship off Scotland dragged its anchor through the undersea fibre cable.
OOOPS !!!!
You really couldn't have made that up (and I didn't).
Alan
A customer was warned that their dual diversity comms links had a potential single point of failure in a national supplier's microwave repeater tower - just for a few months until new capacity was installed. Given that the chances of a microwave tower collapsing were apparently vanishingly small they said they wanted that temporary solution.
During some unusually high winds there was the apparently rare event of just one microwave tower in the whole country collapsing....
It wasn't unusual in the early days of the internet for a UK ISP to be a one-man company with his data centre in his house's garage.
That reminds me of a project team in Luxembourg who realised they could get large discounts on bottles of wine by ordering in bulk direct from the vineyard. The delivery was somewhat delayed as the lorry driver couldn't see anything that looked like a bar in the quiet housing estate for which it was destined.
I've already had someone ring me to ask "was that us in the Reg" so I wasn't as anonymous as I thought...
Someone mentioned quite rightly about the fibre splice time yes - if it was just the car park it would be easy but the JCB had pulled the cable bundle out of the junction box which for reasons past understanding had had a large A-road then built on top of it. It wasn't just us that was offline but the other people had slower copper backup lines going to the same exchange, which of course wasnt the one on fire.
Power bills were a little larger yes but the rack probably only drew a couple of kilowatt as we onlt moved one rack - the rest of the kit stayed in the office and it was only a mail server cluster and the billing database cluster (which took a feed from the mail server so we could bill the clients) that were actually shifted.
As I recall we also had fun towards the end as we were running low on blank CD's - being as you couldnt get these from the supermarkets.
The year was late 1990's and it was the UK to answer the other queries.
Heh! Many years ago an ex-colleague of mine (Hi Stuart) was summoned by the fire brigade to come in and unlock/inspect the machine room in our office due to the fire alarm going off at the weekend.
On arrival, there were no signs of conflagration; however, careful inspection revealed that under the suspended floor there was a pool of water which had shorted out the alarm relays, and was lapping gently at the 3 phase supply to the machines. Turned out that the cleaners were in the habit of stuffing a spare bog roll into the syphon traps in the cistern overflows in the toilets, and that one of same had actually overflowed over the weekend, causing the bog roll to expand massively, block the outlet and start a steady trickle of water running to the lowest point in the building. If anyone can beat "swollen bog roll" as a reason for a callout, I'd like to hear it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
