Re: Nice story
Funnily enough, the worst c*ck-up I've ever encountered with email involved a Linux system. Shiny new Red Hat box, which had all the email from the university academics stored on it, recently transferred from the ancient VMS system. It was IMAP, but I don't recall whether or not it was Dovecot. I didn't administer it.
One day, the email storage got hosed because of some issue with the SAN (twenty years ago; can't remember circumstances now). Oh dear, sorry academics, we'll have to restore everything from backup. In the meantime, they had dialtone mailboxes, so they were receiving new messages.
Go to restore the backup... there is no backup. There was some arrangement where the mail storage was supposed to be backed up via another system mounting the mail storage volume, and this had never been put in place. The RHEL backup was only backing up local storage, not SAN-attached. Oh dear oh dear oh dear.
How was it recovered? Recovering the VMS system, re-migrating the mailboxes to Dovecot(?) on RHEL, and then replaying the MTA logs to catch up the interval between the VMS migration and the loss of storage. Amazingly, it only took a week, although the boss was positively volcanic in demeanour that week.
Conversely, the worst issue I've had in 20 years' of Exchange support was the smallish regional mail server that was happily receiving messages from the MTA and other Exchange servers, queuing them nicely in the SMTP message queues... and failing to deliver them to mailboxes. Since intra-Exchange and MTA delivery queues are different, and the server had plenty of storage, was not over subscribed, each of the email databases were happy and the messages were destined for different DBs, blah blah de blah, trying to find out what was going on was difficult.
In the end, after inspecting logs, checking all services up, stopping/restarting services, unmounting/remounting databases, restarting SMTP, moving mail queues to different partitions... 6 hours later, I gave up and rebooted the box. Once it's back up, BAM, everything starts getting delivered as if nothing happend. All the mail was delivered within 10 mins. THANKS, MICROSOFT!