How many apps does it take to back up your data?

"........continuous data protection a far more realistic goal. The idea is that every bit written by a system would be backed up in (or nearly) real time."

Continuous backups don't help if you need specific, timed snapshots as at end-of-month, end-of-quarter, end-of-year. In these types of backup, you want to stop transaction flow and data updates so that some set of financial and other databases can be copied as a synchronized set. Does the continuous backup software (or service) actually allow for this?

The problem here is

Showing the Finance dept. the price tag of a backup solution the usual response is - "how much?! fugging' ell!". What they fail to appreciate is its about the only software you need to be 100% effective.. hence its worth investing in.

I recently ran into a problem, I had £12k to buy enough licensing to backup 10 ESX hosts at 2 different sites (with 2way replication for DR) and 8TB of backup storage at each.. Upon speaking to Veeam, they wanted £10k-12k alone for just the software - I was pretty much screwed as there was no way I could deliver a workable backup solution. Fortunately Dell and Unitrends came to the rescue, I can't disclose actual costs, but suffice to say it was their end of year and I got 2xDR4100's each with 9TB, and Unitrends licensing to cover all the servers and it came within budget..

Oh boy... I've had to do one (just one) rather large restore, from a combination of tape and disk. It still haunts me in my dreams from time to time, and it's been over 3 years ago. Issue : huge power failure. UPS's went titsup, SAN as well. It took us 72 hours to get everything (AD, a couple of Exchange servers, SQL servers, numerous application servers, file servers, iSeries,...) back online. Most services (not data though) were restored within 8 hours though. And most data was restored by talking softly to the SAN when needed, and giving it a kick when it wouldn't listen. Nerve-wrecking (check spelling Trevor !) indeed.

When afterwards we went to management to discuss RPO/RTO they didn't really care that much anymore. It was fixed.

Nowadays we use Veeam to handle Windows servers (we're on vSphere for 90% of servers) with replication to another site, and tape backup to a remote site for iSeries. RPO is still 24 hours though, RTO best effort. Selling decent backup procedure/solutions to management is bloody hard, even when you've looked disaster in the face and kicked its ass.

Reconstruct, don't just restore

Your backup administrator fundamentally needs to decide whether they are backing up state or reconstruct-ability. There are many tools to backup state, but they tend to evolve into fragile silos: you have your database backup, say, and then your web server farm, each has its own backup tools and saved-backup format, and you can't say that service is back until all parts of all silos have been restored. And thrice woe unto you if you missed out something that should have gone into your silo at the time of the last backup. (You mean the special sauce config wasn't in the Registry? And the app doesn't start without it? Err.....)

Reconstructability is a different approach. Supposing your systems died and your backup tapes vaporized. How would you rebuild everything? Maybe you would extract data from a payroll system, or your web designer's CMS, massage it, and then squirt it into a database. These are the steps that need to be documented and automated. How will you rebuild the database in the first place? Maybe you'll use Chef or cfengine or somesuch. The point is not to attempt to capture state but to actually recreate what you need to get up and running.

There probably isn't a 100% correct answer for all scenarios, but over the years I have become more convinced that reconstructability is much more important than state backup, and has many positive benefits outside of backup/recovery besides.

Management don't like IT in the loop

I was an IT Admin for a small company in London. They had a NT server with a poorly maintained RAID5 array and tape backup system that stank (when I joined that is). I nursed it along and tried several times bringing up the subject. I was never invited to management meetings. One day one of the disks died.

I thought we was OK because the tapes were there if another died during a rebuild. Spent all night fixing the array. I then spent some time checking the tape backups only to find most did not even verify and there was a problem with the backup regime itself. The entire company was hanging over a cliff holding on by the skin of its teeth. If any other disk had died during the rebuild it would have been gone.

I put all of this before management in written form. It was ignored.

Several months later due to poor health and stress I was made redundant. My last words leaving that company was 'sort your IT out'.

I tried searching for the name recently. Either they changed it or went under.

Management can be pretty thick when it comes to IT needs.

Depends on size and usage

For small businesses without 24 hour operation - full image backup overnight (or at weekend) is probably the best option as it is certain to have a consistent state of all applications.

For small to medium businesses with 24 hour operation but small number of servers, full image backup using one of the disk snapshot products is probably the best - note there must be sufficient spare disk capacity to handle the requirements of the snapshot software. Depending on the business there might also be a need for transaction data to be copied offsite if loss of data since the last backup cannot be tolerated or recovered.

For large businesses with a number of servers that need to be kept in step - this is where the problems occur - there is no "one size fits all" solution. Full mirroring at a remote site MAY be possible but expensive. (Dedicated fibre links between the sites would be required along with duplicate disk arrays etc.) If the site is big enough then having the DR system onsite but far enough away to be safe may be possible (but again expensive). Trying to get a consistent snapshot across multiple active servers tends to be DIFFICULT (understatement). If a person (or team) is given the job of designing the backup regime in this case then the first thing to do is to try to find out what data is fairly static and what changes frequently. Then find out from the business teams how important consistency is - for some data it will be critical - for others not so important.. Decisions need to be made on what should be done if the backup system fails (e.g. Mr JCB digs up the offsite fibre link).

If backing up to "the cloud" is considered then legal problems may occur unless the data is heavily encrypted due to the collision between the EU data protection directive and the US government slurping up the data that is sent over the internet or stored on a US cloud server. Cloud storage is also expensive when the network costs are added to the storage costs. (Just to backup or restore a single full 1TB disk in a 10 hour overnight backup window requires an uncontended 230Mbit/sec link !!!)

Product placement much...

Trevor - how much did Asigra pay for the skyscraper ads on this page?

I do not believe that a single backup application is the best because it does not not know the specific gritty application details however a general purpose backup application is great for all of those miscellaneous applications. Use an OS specific backup app to backup the OS because there seems to be hidden files and files with extra "this file cannot be read". Database have to have their own backup software to ensure transactionally correct backups.

Oh yea, also make a separate small "in case of emergency" backup which can be used to create a minimal system to start the recovery process.

A multiapp method is way more work but the single applications do not seem to have the documentation that I would like see and understand that the application provider has thoroughly tested the app. For example try to get a list of files which the backup application does not backup.

If you are using only one app

Don't use backup exec 2010! I was platespinning our ISA box to a Xenserver VM when I got distracted and uninstalled ISA from the live VM. Thought ah tits but oh well it was backup up last night. Did a full recovery and half the ISA files were missing! Not one error/ notification on BE2010. Luckily could export some config xmls and got it back in 90mins. Before I left the CFO must of been high and asked for SAN>SAN rep at our other site. Other guys in the biz I know have not been so lucky. Especially after getting a new Farm of DL380s after 2 years as the last twat bought ones that were gimped to 32GB ram per host!