Have a Plan A, and Plan B – just don't go down with the ship

Plan A

* Deny any and all responsibility. Shift blame as appropriate.

Plan B

* Have my CV always up to date.

Plan C

* Go down with the ship. :-(

Active-Active

Worth considering the use of an 'active-active' system where two duplicated systems are both active with delta's applied between them to keep the databases in sync,

Quite common in the payments industry.

If you lose one system at least 50% of your terminals/access points are still working and your database is fully intact and up-to-date without needing any manual intervention.

You can then manually (or automatically) swap the remaining connections to the still running site.

You will still need well documented procedures and processes but it can take some of the panic out of recovering,

Plan 9 (from the German class)

Mach nur einen Plan,

sei ein großes Licht!

und dann mach 'nen zweiten Plan -

gehen tun sie beide nicht...

- Bert Brecht

I worked for a large insurance company in the early 2000's. The site I worked at contained the mainframes that supported a significant chunk of the consumer financial product and pensions side of the business and around 2500 staff.

They decided to introduce the IT department to the non IT (personnel and logistics) aspects of disaster recovery by talking through a scenario where a huge storm was forecast and the site was subjected to local flooding and high winds. Power and phone lines went down and staff were unable to leave the premises due to fallen trees.

That this was the first time the company had thought about this was obvious from the start.... because the first part of the site to get wiped out was the backup tape store which was in a small well secured enclosure on the edge of the site, right next to the river and surrounded by tall trees.

There was no offsite backup....things went downhill from there!

Don't forget UPS arrangements

How many of you have pulled the Big Red Knob on the master switch to the building/campus to see what really happens when the mains fails for more than a second?

Do the UPS hold up the machines but not the A/C systems?

Is there enough emergency lighting and torches (in working order) to get around and do stuff like check the power outage is not one of your own breakers tripping on a now-cleared fault?

Many years ago I worked for an insurance company which had a DR plan in place. Since I worked the Wednesday through Sunday swing shift, I got to participate more than once.

The drills were random and could happen anytime on a weekend. They were used to test the complete recovery and uptime of the systems we were using at the time. The mainframe system was offsite since the beginning so all that was needed for that was a network connection.

We sent our tapes to an off-site storage firm. Initially it was a small company which failed the test only once and lost the contract. After that it was a much bigger company who came in daily, including weekends, to pick up the nightly backup tapes from the previous night's backups.

During the tests, we would get a call, that the systems were down. Being in computer operations in the computer room, we would find ourselves in a locked-down type of situation where there was no local network access as someone had pulled the plugs to the concentrators.

The system at the remote location was brought up on line once the tapes arrived and the tapes were restored to the servers. The systems were then brought up online and we were then told to connect to the system and run the batch jobs we were scheduled to do so, except they were running off of the remote location. Everything went as planned.

The only glitch we had was that offsite company who messed up and didn't respond to the phone call.

Granted this was 25 years ago now and stuff was a lot simpler. The servers were '486 Novell servers and the tape backup was perhaps two tapes maximum at 1.2Gigs each.

What's interesting is how this went together like clockwork and how well everything was rehearsed to ensure it did every time. We were usually up and running within a couple of hours.

I have always maintained good backups of everything, even at home as well. The important thing to remember is hardware can always be replaced, software can always be reinstalled from media or these days downloaded off the web. The data is the most important thing to backup, and without a good backup of the most important thing, you have nothing.

Most important thing:

Have management who continually ask themselves if they have the staff who can do all this plan A plan B business, and ensure that said staff can demonstrate they're capable of it.

That's the hardest bit, because none of this huge effort has anything to do with the ordinary day-to-day running, and it takes a leap of imagination that is often sadly lacking among upper management...

Where is your disater recovery/business continuity plan?

I used to work for a major bank. We had hot standby systems, network connections, automatic fail over and everything else that was needed for the major systems. The DR/BCP plans were tested on a regular basis including running for periods of time on the backup systems then initiating controlled switch over to the primary systems.

The problem was with some of the "lesser" systems (one inventory management system in particular comes to mind) that ran in locations other than the main data centres. We had a "disaster" and the first question was "Where is the DR plan?". The response from the sysadmin - "On my desk". Consequently we had to "wing it". Fortunately the people who ran the systems on a day to day basis were available otherwise we would have been in deep SH*T. After that we made sure that the DR plans for all systems were available in both the primary and backup centre (and when it became more available - on-line) and that there were staff on site all the time in case access to the plans was required.