"I almost pulled the wrong drive out of a raid 5 array once on an on-site many years ago. You learn a lot from almost mistakes."
Arriving at a customers site at 7:30pm (yes, it was that urgent, out of hours rate) to replace a drive in a three drive array, there was only the security guy left on site. It was so important that no one with any authority bothered to stay behind to make sure everything went to plan. But someone had kindly pulled a the faulty drive out for me. Except the LEDs indicating activity or fault were to the LEFT of the drive, not the RIGHT.
I left a written note, countersigned by the security guy, telling them that without username/password or knowledge of their system I was unable to even touch their system without further authority in case of any data loss etc No way in hell was I putting the good drive back in and swapping the bad drive out in the hope that it might be alright.
When they called my boss in the morning, screaming and ranting, he told them in no uncertain terms that they were in the wrong and he'd be happy to terminate contract once they'd paid for the call out. Apparently it all went very quiet, before they agreed I would come out again and work with their IT guy to swap out the HDD once he had brought the degraded system back up.
Personally, I still to this day think there was something else going on with their system, something catastrophic, and we were being set up to fail and end up paying for data recovery or some such because theoretically it should have been a simple case of replacing the good drive and bringing the system back up with the RAID in a degraded state, at which point replacing the bad drive would instigate a rebuild. Or the IT guy had screwed up and was trying to cover his arse.
Oh, and once the bill was paid my boss cancelled the co tract with them.