reboot of terror
I was working for an MSP in Orange County, CA. One of my responsibilities was to configure new equipment that was being shipped to a customers retail locations. The engineering team made this stupid easy in so far as open ticket in system > click config > copy config > paste to ssh terminal > write config > reboot and verify that it is communicating with the headend.
I was configuring something in the area of 10 boxes and I was starting to close in on the shipping window. I had them all stacked and I had all the boxes open in SecureCRT with the headend as the first tab. Pasted and wrote all the configs and no issues. Started issuing reboots to test and, sure as shit, I reboot the headend.
Normally, for our customers, this is not an issue because they usually have redundant headends, not this one. Open a console, start pinging public IP which is whitelisted to respond from our site
Go talk to the engineer who is packing up to leave, fess up to what I did, the broken man puts his stuff down, opens a console and picks up the phone. I ask if I need to stay and he says no.
The next day the engineer is absent and I look up the customer, our ticketing system at somewhere around 3 AM had autogenerated and autoclosed somewhere north of 3000 tickets. Come to find out from another engineer that the headend in question had been running with bad nvram so the reboot wiped the config and there had been no IOS to load. The customers engineer had discovered the headend sitting at a ROMMON prompt. New IOS and new base config later and the customer is back up.
In the end, I was told everyone gets to do that once.