Speaking of Financial institutions
Many years ago we were running a very small options desk and running off a single server. When the time came to upgrade, we had to do it overnight, and by 4.30 am (after starting at 6am the previous day) I was more than a little groggy and decided that it was time to call it a night ( and crashed out on one of the couches in the board room). Next day with all the stress, I forgot to copy the historical data across to the new server. A few days later, one of the other admins asked if they could re-use the server and without thinking I said yes. About a day later I was asked where all the historical data was :(
Unfortunately they needed it for a court case and the only backups were done on DAT tapes using a reader in the old server. So after a Stat Dec saying that the data had been irretrievably lost (We spent ~$5k on a data recovery service but were unable to recover the data) I learnt a couple of things.
1. document the upgrade process and test run it before proceeding.
2. make sure you have a checklist for data checking and completion.
3. have a roll back position (much easier with VMware)
4. retain the old server for at least a month after the process.
5. never do a job like this when sleep deprived