monitoring software issue
At this WW DC, we were using VPO (ex-ITO) as the HP monitoring solution for the whole DC.
One sad friday, an idiot from HP lasted his day by selecting 90 000 old alerts, clicking "delete", then fucked off for the week-end. Of course without telling anyone in the ops team.
The dude on duty didn't really appreciate spending day and night in the WE, troubleshooting the fact the DB under VPO would crash like every minutes, due to logs moving to archives at insane speed, filling up the file system.
Of course, the workstation with th VPO GUI was not remote accessible ...
Monday saw a bunch of angry emails, unsurprisingly.