back to article Forget your RTO*: Real world Disaster Recovery needs garbage bags and bubble wrap

If it's Friday morning it must once again be time for On-Call, El Reg's end-of-week meander down memory lane to explore readers' reminiscences of jobs that went bump in the night. This week, reader “Olaf” shares his experience in Thailand's 2011 floods. Yup, those floods that left Western Digital's hard disk factory under …

  1. Will Godfrey Silver badge

    Well Done

    ... and there was me feeling hard done by when I had to step through 2 inches of water in a factory some years ago.

  2. Anonymous Coward
    Anonymous Coward

    Leaky roof

    Ah, our nice (relatively new) office building with a roof that tends to leak any time it rains too heavily - cue water running down the walls/pooling of the false ceilings/coming down through into the comms room straight onto the UPS kit. Aged a bit when I opened the door and saw that I can tell you :(

  3. Ryan.T.Student

    My main memory of Thailand was a spell as an IT teacher out there.. in the computer lab it wasn't uncommon to have to reach under a desk and mess with a computer. I got an electric shock every single time. Trying not to swear in front of a class of 5 year olds is quite challenging.

  4. Marc 25

    If, in your career, you've never had to rescue a server or backup media via means of a makeshift boat. You aint lived!

    4 times in my career i've watching gleefully as my place of work has sunken beneath the waves of the various rivers of the UK. All of those businesses recovered with very little data loss. All of them continued operation and still operate today.

    First rule of IT: Have you backed it up? Do you have 3 copies, in two locations, one offsite?

    It's amazing how many people don't plan for the major disaster situations, fire, flood, earthquake, nuclear attack or even war.

    1. allthecoolshortnamesweretaken

      I take it all your gear is EMP-hardened, then?

      1. Marc 25

        I take it all your gear is EMP-hardened, then?

        Nope, it was geographically placed to minimise political impact

    2. Intractable Potsherd
      Joke

      "4 times in my career i've watching gleefully as my place of work has sunken beneath the waves of the various rivers of the UK."

      I think you need to change your username to "Jonah" ...

    3. Stoneshop
      Windows

      Sunken beneath the waves

      Everyone said I was daft to build a datacentre on a swamp, but I built in all the same, just to show them. It sank into the swamp. So I built a second one. That sank into the swamp. So I built a third. That burned down, fell over, then sank into the swamp. But the fourth one stayed up

      1. John Brown (no body) Silver badge
        Thumb Up

        Re: Sunken beneath the waves

        "But the fourth one stayed up"

        Yeah, because now you have 40' of foundations on top of the other three :-)

        1. Daedalus

          Re: Sunken beneath the waves

          You're commenting on an old joke.

    4. Anonymous Coward
      Anonymous Coward

      > It's amazing how many people don't plan for the major disaster situations ...

      Especially those tasking others to do it !

      With a previous hat on, we got a directive sent down from on high (well OK, from our director) instructing us that we need to "do" disaster recovery plans for the IT stuff. It came out of the blue, and you may note the "isolation" inherent in the "for the IT stuff" bit.

      AFAICT, it had come down from above - either our parent company or an insurance company - and so the clueless that led us simply passed it on.

      Now, I had a bit of an idea what this might involved - well really I had enough idea to know that I didn't really know all that much. I asked around on some mailing lists, and it transpired that someone had a vacant seat on a business continuity (BC) course the very next week, and since some of the course involved role playing games, he would let me attend the course for little more than what it cost him for our accommodation (it was a 3 day course) - with the proviso that I didn't let on to anyone else how little we were paying of course !

      It was something of en eye opener - and confirmed how little I'd known beforehand.

      Anyway, I got back to the office, and started asking manglement about the business requirements - because you don't "do" DR for the IT stuff without input from the BC plan about what the business needs are. The conversation was over in two sentences - the second one being "Don't be awkward, just do it" from the director.

      As it happens, not long after that we had a fire alarm - not a drill, but it was a false alarm. So there we all are, stood outside in the car park getting very wet - actually, I wasn't getting very wet, I had the presence of mind to grab my car keys as I left my desk and could get a brolly from the car :-) I casually asked one of the directors what their BC plan had to say about welfare of the staff - given that most had left their belongings in the office, they couldn't use their cars to get home, some would not be able to get in if they got home, did they know who held the keys to the village hall, etc etc. Is there a school where upper manglement get to learn that particular expression of "humff" they use when some impudent junior asks an awkward but quite reasonable question they aren't going to answer ?

      Just to add, the office was alongside a railway where occasionally nuclear flasks get to travel - so whilst not "nuclear war", an exclusion zone preventing access would not be an unforseen thing to plan for (if, for example, the train derailed).

  5. Sureo

    Olaf should get his PhD in moving IT gear.

  6. john bertelsen

    Our state IT workers were quick enough to move server out of the basement of the state office building in Waterbury, Vermont as Hurricane Irene poured rain into the nearby river. Some of them lost their cars in the parking lot, but they saved the servers....on a weekend.

    Much of the central part of the state was paralyzed for weeks and the state workers did not get new digs for several years, but at least they were able to operate.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like