back to article Ford slams brakes on sales spreadsheets after fire menaces data center

Ford is working to recover after a fire at its US corporate headquarters briefly shut down data center operations and prevented it from gathering sales data. The automotive giant said that the Monday fire, which broke out at a power sub-station beneath the building, caused a full evacuation of its headquarters in Dearborn, MI …

  1. Dwarf Silver badge

    DR done right

    Nice to see an example of DR done right.

    Having said that, there seems to be a trend across data centres of power capacity management not being managed properly. I recall a couple in Telehouse earlier in the year where the magic smoke was released.

    Perhaps its time for the enterprise management platforms to have a view of power consumption and power limits. Also sounds like more could be done for regular capacity management on the critical power sources to the data centres.

  2. CheesyTheClown Silver badge

    Re: DR done right

    Did we read the same article? This was a piss poor example of disaster recovery. All I could think while reading the article was "Sounds like Ford".

    Any company managing their own servers should have a minimum of 3 data centers spread out geographically. Their systems should have 100% (not 99.999) uptime and they should be thoroughly embarrassed by any announcement of this type. If I were in PCI enforcement or banking regulation enforcement, I would open a case to investigate gross negligence.

    Ford should really outsource their data center to someone competent with technology. They have proven for nearly 100 years that anything with electronics designed by them is going to constantly suffer failures.

  3. Anonymous Coward Silver badge
    Mushroom

    Re: DR done right

    They experienced a major disruption and that resulted in an ACCEPTABLE level in interruption. That sounds like DR done right to me.

    Note: DR, not BC.

  4. Anonymous Coward
    Anonymous Coward

    Re: DR done right

    Given cloud providers don't get 100% uptime (eg http://www.networkworld.com/article/3020235/cloud-computing/and-the-cloud-provider-with-the-best-uptime-in-2015-is.html), I don't see why that's a relevant level. You should provision your availability to the appropriate level for the cost of an outage, not to perfection just-because.

  5. Old one

    Re: DR done right

    Last I knew Ford's headquarters didn't keep track of people's medications, fire dept staffing, hospital staffing, air traffic control, the greater Dearborn power grid, US DoD communications, or the internet. So considering the job to be done, seems it did just what it was supposed to do.

    And remind me but wasn't Ford the ONLY major automaker that did NOT take any Federal bailout money? Wonder if that was because Ford management knows how to MANAGE their assets.

  6. Anonymous Coward
    Anonymous Coward

    Re: DR done right

    @Cheesy

    Think you can do better? Better get your CV in quick...

  7. Fungus Bob Silver badge
    Trollface

    Re: DR done right

    Who is this Doctor Dunwright?

  8. KX1B

    Great to hear DR works

    Rarely do you hear of a DR event that works to save the enterprise from catastrophe. The CIO should be praised for ensuring a fire didn't cripple Ford

  9. smartypants

    Are they still running core memory?

    "with battery backups on-site preserving all stored data"

    Also...

    "Nice to see an example of DR done right."

    It's worth pointing out that the facility is offline. This is better than on fire, and better than 'all our data was lost', but it's not business as usual, and it sounds like all they're doing is waiting for the power to be restored, so I'm not sure that counts as 'disaster recovery' in my book. Even the worst IT setups I've seen would probably survive this.

  10. Dwarf Silver badge

    Re: Are they still running core memory?

    @smarrtypants. The article says

    The automotive giant said that the Monday fire, which broke out at a power sub-station beneath the building, caused a full evacuation of its headquarters in Dearborn, MI. The fire also resulted in power being cut to its on-site data center that was scheduled to receive sales information from Ford dealerships across America.

    So, a mains power failure occurs and the UPS maintains power whilst they work out the problem and decide what they are going to do (wait for power, generators kick in, initiate DR) - exactly like the system would have been designed to do. Success story #1

    A Ford spokesperson told The Register that the data center itself is intact, with battery backups on-site preserving all stored data and remote facilities able to shoulder the load while power to the Dearborn datacenter is restored. The company said that no other facilities or services have been disrupted.

    This implies that they have invoked DR and the "remote facilities" will be the replica DR location doing its job as designed. Success story #2

    This is text book stuff, IT working exactly as its designed to do and the primary reason for spending big piles of cash on a situation that will probably never happen. Its just insurance in a different form.

    I'd like to hear more about "the worst IT setup" surviving this with the lack of power, UPS or a secondary DR location. Please explain how you expect this to work. After all, there is a well known truism of "it works better when its switched on" which seems very apt for this article.

  11. Nolveys Silver badge

    Earlier that day...

    "I guess you're the new guy, eh? I'm Frank, I'll show you the power substation."

    (Frank opens the door to the substation, inside is shrouded in darkness)

    "For some reason you have to walk across this room in the dark to get to the light switch. I'm not sure why the - OW! MY DAMN SHIN!"

    (With a click the lights turn on, revealing what Frank has walked into. It's a Ford Pinto)

    "How the hell does this God Damned thing keep getting in here?!"

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018