Boss hands dunce's cap to chap who turned off disk monitor

Yup been there

Some bright spark in one place decided to disable BMC alerts as they generated too many incident tickets and made his stats look bad - can't really see how replacing them with one, big high-sev incident was any more desirable but hey-ho!

Dont shut the server down via the os, let me remote in and do it. Followed by 15 hours lf lost data whilst it rebuilt following an uncotrolled shutdown. Why pay for support and then ignore our intructions!?

Network is down

Well RS485, so just about describable as a very slow network. but, hey, it's been there since the dawn of PCs, and runs reliably 24/7. System directs plastic granules from silos to extrusion machines, controlled by an ancient computer. Remotes are just dumb relay drivers. No such luxury as remote login, so yours truly had to drive there in the middle of the night. By the time I got there the factory flour was covered in plastic granules - lethal! The operators had eventually managed a sort of shutdown and closed off all the silos manually.

Computer simply said "network failure". I went to unplug it from the computer to do a basic continuity check and immediately realised it was plugged into the wrong D9 socket. Put it back (into the shiny clean one) and everything sprang into life. Apparently one of their guys had decided to 'tidy' up the wiring. Everything just continued to obey the last command it had received, so it was a while before, one at a time the machines over-filled.

Disappointed

It was a metaphorical cap. I was liking the mental image of the person who cocked up wearing a papercone on his head in the company of their fellow workers.

Possibly more effective than a formal tete a tete with manager or black mark in their personnel file.

You can set an Exchange server correctly? How? How? How?

rm -rf

No, not *that* classic use of rm -rf, but..

The Financial Director where I worked a bazillion years ago needed to restore from tape the (b)ought ledger files to our SCO Unix box.

Unfortunately, he restored the files to the root of the server's disk, and rather than move them into place, he restored them again - correctly - and then issued rm -rf b* from the root folder.

Things were hunky dory for a few weeks until we rebooted the server during overnight routine maintenence and it didn't come up again.

One trip to the office later and I'd discovered that /boot was missing. Two hours pass and we were OK for the morning, but I had to wake up the MD to open the main office where the master tapes for the OS were stored in the safe.

Oh yes, been there

We used to have a 200MB file of random data located in the root of an older, vendor specific unix, so that when the application numpties got a path wrong and filled up root filesystem, we could run a backup with a pre-script of "rm [temp file]", then rlogin quickly and kill the relevant process (This was when a full root booted you out of ssh, where the backup agent process were already spawned and could run).

or the time the trainee operator wanted to learn unix, so added his userid without a home directory, which defaulted to root /, then did a "userdel -r [userid]" to clear out his "home directory".

Let there be light

It was a Saturday morning; I had woken up, but was luxuriating in a warm bed, thinking about 2 whole days off to do nothing. Then my mobile rang and I answered it; big mistake.

The manager of a site some distance away was almost screaming down the phone at me to help sort them out; no access to email, no access to data etc. etc. Tried to remote in but no response. I did ask if there had been a power cut and he confirmed that there had, but everything was now back on.

In the end, I drove there, about 4 hours through some shitty traffic. When I arrived, I found out that the local electric company had been doing some work to the power transformer nearby; the site had been informed of this, but didn't think to tell anyone in IT. As a result, when power went off, it all went onto UPS; an alert was sent, but unfortunately, the router had failed before the email was transmitted due to not having been put on the UPS by the staff member that set it up. Plus the UPS had drained down so far, it shut everything down.

Even worse, when the power came back on, it caused the breaker for the UPS circuit to trip, so nothing on that line restarted. It took me about 10 minutes to identify this, restart some of the kit, then I had to hang around for a while checking everything cam back on.

After that, another nearly 4 hour drive home. All because some git put a power lead in the wrong socket

A friend told me this story a few years ago.

It was back in the days of Windows 3.1, so Windows actually didn't do very much to protect itself against abuse..

The aforementioned friend worked for a small budget software house, and they were working on a football game. They had a reputation of development cheap, low budget game. To give you an idea of the kind of money they spent on development, the soundtrack to each match (which the packaging made a big thing of it being digitised) consisted of the CEO and his PA cheering, then shouting "Palace" in one corner of his office. That sample was used whether Crystal Palace were one of the teams in the match or not.

Apparently, the CEO called my friend one morning to say his machine wouldn't start properly. When my friend got in (he couldn't drive, so had to get a bus which took a while), he booted the machine from a floppy. As far as he could work out, the boss had stored all his documents in the Windows folder on the C drive. He'd decided to have a clear out, and someone showed him the "del *.*" command, so he happily ran it on the Windows folder. Obviously he had no backups of any description.

Call out fun

Good call out stories here, reminds me of past call outs. Being on call was part of most past jobs. Often I was called out more than others and was usually called for the most difficult or expensive problem. Not because I liked being called out, just the opposite. I hated it and would try to fix problems over the phone and always created a workorder or trouble ticket to fix the cause so it wouldn't happen again.

Which is why I figured I got a call at 0300 to be at the airport by 0500 to be flown by helicopter to a remote site hundreds of kilometers away. I'm tired so go into auto mode, get up, get dressed, and then I realize I don't know what tools or parts to take because I don't know what the problem is. There is no one at site so I call the control centre and ask them to look at their screen to see what the problem is. They look and say it is all good, it's still there.

I ask what he means by "still there" and the operator explains there has been a threat against the site that it is going to be blown up or something but obviously it hasn't been bombed or anything as there would be alarms. Then he says not to worry there are "people" going in at 0500.

A few more calls and it turns out I was the "people" (no kidding, just me scheduled) so I call my boss and ask for authorization to take in extra tools. What tools he asks? I give him a quick list which includes my Mini-14.

That results in more phone calls but to make a long story a bit shorter yes I was being sent to a site because it had been threatened as it was felt I was the best choice and would be able to handle whatever was going on or at least report back what was required. A nice complement and probably true but a heads up would have been nice.

We did eventually get more training which included dealing with bombs and threats. That didn't prevent a nearby site being taken and workers held hostage and another interesting call but that's a whole other, very long, story.

Being a Tech doesn't always mean boring.

Hard drives get hot too...

I used to work for a corp. that developed voicemail software.

Back then, the largest hard disks were 512 MB each, 5.25" drives that were several inches thick and gave off a lot of heat, and they required mapping of the error blocks, so we kept the printouts with the computer that housed them so we could reformat the drives if necessary.

I was in the QA lab late one night when I got a call from the CEO that the in-house voicemail server was down - would I go look at it and see what the problem was?

I went to the server room (a broom closet re-purposed for the in-house voicemail servers) and looked at the production server. I found that the person who had setup the server had stuffed the printouts of the error blocks between the hot drives and the drives had overheated and eventually started having errors. I took the server down, pulled the drives, copied as much data off them as I could onto other drives from the QA lab, and replaced the drives. I was there until well after midnight.

A lot of people lost a lot of voicemail that night. I don't recall just who had setup that server, but I sent out a company wide email explaining that while it was a good idea to keep the error block printouts with a computer, it wasn't a very good idea to block the airflow in a computer, especially in and around physically large hard drives that give off a lot of heat. We were lucky there wasn't a fire.

Had a call from a company who said their NetWare 286 server (the one with the grid display?) was down. I asked them to check it was switched on and they said yes - they could see the display. Got to site and noticed that what they could actually see was the grid burned into the monochrome display and the server was actually unplugged. Following the extension lead that had been plugged into the servers plug socket, the culprit was found drilling holes in an adjacent office.

I got called out to a British Rail office in Croydon who couldn't login to their Torus server. They'd had new carpets fitted over the weekend and the fitters had cut the coax cable at each floor point.

I remember a guy who used his IBM ATX with a missing drive panel as a useful ashtray.

I got a call from an angry user whose Wang Pro was making a continuous beeping noise. When I got to the office it was sitting on the pavement outside after leaving the building via a first floor window.

Kettles & Toasters & other things that trip mains RCDs.

Once we'd grown large enough to require something more organised that "floppy-net" we installed a DEC P60 Novel Server.

Locked in the room that served as the warehouse, hardwire into a switchless socket so no chance of accidentally turning it off or unplugging it.

Feeling pretty smug about power we couldn't understand why the server kept shutting down.

Turns out it coincided with tea & toast time. The hardwired "server" power was on the same ring as the kitchen.

New ring installed, UPS purchased.