I once was caught by a network outage whose ultimate cause was:

An X509 certificate, in a Java package, had expired. The Java package was part of a UPS monitoring software (and nothing you'd ever normally play with). The UPS monitoring software went absolutely crazy and just hung the entire machine (as in, your mouse couldn't move across the screen in under a minute). The machines affected were pretty important and all of them had the problem. Because they still responded to some queries, pings, etc. (just about) nothing bothered to failover anyway but even the failovers had failed. And you couldn't log into the damn things or use any of them directly.

And obviously, the entire network slowed to an absolute crawl affecting the entire place it was in. Sure, we could force failover to a clean machine but those servers had apps running and we needed those apps (data - pish, we had that, but if you can't submit the payroll without that heavily-locked-down smartcard tied only to those machines you were stuffed).

Sounds like nothing but go through your standard procedures - everything responds, everything responds to ping, just some things are very slow. Can't narrow down to one server. Try to log in to them remotely and it timeouts. Go find the physical machines. Try to log into them. LITERALLY could not get the mouse to move. Suspect hung servers. Wait suitable length of time. Force a reboot. Server boots up at normal speed, does disk check, nothing wrong, gets to login screen, hangs like hell.

Okay. Weird. Safe mode. Same thing. Okay. Disconnect from network. Same thing. Restore from latest backup (which takes HOURS). Same thing. Okay, restore from Known Good backup (more hours!). Same thing. Confirm the machine is fully operational with install from clean Windows CD (but too much software on there to just replace every server in the space of a day or so from nothing). Perfectly working, no problems at all. Okay, restore from ancient backup taken just after the Stone Age and which DEFINITELY, DEFINITELY, DEFINITELY works. Same thing. WTH? Try other machines. Same thing.

Restore a backup to an identical machine acting as tepid-spare. Same thing. Bugger. Complete reinstall looking like only way to get it back and working. Clean Windows install on clean, isolated machine. Start reinstalling software from disks (known-good versions of everything that worked for YEARS). The reboot AFTER the UPS software is installed starts displaying the same symptoms. Damn! (But at least we can narrow down). After a couple of installation retries (and rebooting after literally every step), narrow it down to the UPS software installation. Update available? No. Reported problems? Yes, with same symptoms. No solutions.

Fall back onto the last resort of IT: Phone random IT people and see if they've had the same thing. Couple of them experiencing it, one of them in the advanced stages of diagnosis. After much jiggery-pokery and literally just turning the software on under heavy debugging/monitoring (and LOTS of rebooting) and trying to identify cause, narrow it down to accessing a certain Java package and then, inside that, find an expired certificate. How do we know it's the cert? Can you fool it by putting the clocks back? Actually, yes. Before the expiry time, it works, after it doesn't. Can you replace the cert? Yes, but the software refuses to work even if it doesn't hang. Can you fake it? No. Can you remove it? No, the software won't work. Can you run your processes with an inaccurate clock? No. Can you run your server without UPS monitoring? Not really.

So it takes this long to get to knowing what's happened and what you can do about it. The affected servers can have their filesystem accessed by something else (Linux boot disk) and have the UPS software disabled, because there was literally no other way to get the software turned off once Windows had started (and actually, I moved the UPS monitoring to a Linux machine that then would issue shutdown notices to the Windows servers instead!).

Eventually, after about a week, the UPS manufacturer issue an update and pretend it never happened (and NOBODY installed that update without first installing clean or working out what the problem was). But you try diagnosing that bugger while having a whole company breathing down your neck and silence in the server rooms.

What saved the day? Linux, contacts, careful investigation, no blind reinstalling, and not relying on the UPS manufacturer to actually DO anything about it.

The last thing you expect when your whole network goes down (and only the servers, not the clients) is some UPS software that's been working FOREVER, on known-good versions, on known-good servers to suddenly stop working even on restoration of older, working backups (whose restoration had ALWAYS worked before that point when testing backups) and then hang the machine to the extent that you literally could not do ANYTHING on it at all. I was just about ready to kill Microsoft until we saw that no updates had taken place in the weeks leading up to it and the servers had been rebooted and backed up and test-restored since then.

Not a sys admin but...

I've done something which might just about qualify - it's more hardware related than admin related to be honest. Actually, it's stupidity related. And as I'm not a sysadmin, I'll remain anon and discount myself from the prizelist.

I was, err, playing kinda "bit of stick and a few rolls of paper cricket" in the server room with a colleague (yeah I know, I know...)

My colleague scored at least a 4 as the "ball" sailed over my head and went behind a cabinet. In my hurry to ensure it didn't become a 6, I rushed round the back of the cabinet. And not just any cabinet, oh no. This was the cabinet where every terminal in the entire organisation had an RS232 patch lead going from the back of the racks into a patchboard on the wall.

I tripped, and landed in amongst all the cables. Very few of them had covers, just three wires soldered onto the pins, and most of them were a pretty tight fit into both the router-thingy (I can't remember what it was called) and the patch-board. So it just ripped the wires straight from the pins. I must have destroyed about 80 or more cables.

We had to sit soldering for at least 4 hours solid.

To add to the fun, none of the cabling was labelled, so we had no clue which machine-connector went to which patch-connector, so we were plugging people into someone else's still-logged-on account right left and centre.

You might imagine the first thing we did was panic. Wrong. The first thing we did was get rid of the bit of stick and the rolls of paper. Then we panicked.

During my first week as an ICT Techie at a school I encountered a problem with MS Office on one of the machines. Reading round a few of the knowledge base articles and support tickets the answer was to re-allocate the MS Office package to the workstation, basically un-installing and re-installing the whole of Office. Being as I'm such an efficient person I thought "Heck, why not re-allocate the package to the entire domain therefore fixing any other machines that may have the issue but no-one has noticed yet". Fantastic idea, especially 2 days later when people could finally use their computers again as it brought the whole network to a crawl.

First week in a new job not knowing enough about what your doing is a bad combination.

On the other hand, Office worked pretty well after that.

---

I also recently had a colleague panic that one of the displays on a laptop had packed in, he went through the usual processed including contacting the manufacturer to check it's still in warranty before someone said "Have you tried the key combinations to increase the brightness" to which he replied "Of course I have, but strangely it's just started working fine" - hmm...

Cheers

Lights, camera, silence

First week running the bank branch on my own. Walked in as usual one morning to change the tapes, turned the lights on, and had the gut churning chunk sound of every machine in the server room losing power at once.

Several hours of mainframe IPL later and we're finally back up with me on the phone to our electrical guys to work out how the hell the room lights were wired into our UPS power.

Everyone assembled onsite the next day to prove that it wasn't possible. Took everything down just in case and brought them up in standby. Most satisfying junior admin moment ever when after 3-4 attempts by the sparky I walked up, hit all four switches at once and the room promptly crashed to dropped jaws from all inside.

Turned out to be a bit of dodgy wiring in an unlabelled sealed over switchbox in that section of wall - it was an old power cutoff wired under the floor to the master cutoff switch on the far side of the room. Since the wall was just a cheap partition, the vibration of the door closing over time had jolted a wire loose, and turning the lights on at once was just enough vibration to make the wire jump free and take everything down from inside the UPS supply.

Good lesson that even completely disconnected systems can be linked, if not in the fashion you would have expected.

Our VPN box...

[user@server ~]$ date;date;date;date;date;

Thu Oct 21 10:12:23 BST 2010

Thu Oct 21 10:12:18 BST 2010

Thu Oct 21 10:12:23 BST 2010

Thu Oct 21 10:12:18 BST 2010

Thu Oct 21 10:12:23 BST 2010

Disabled anything remotely to do with ntp. Rebooted. Didn't make a difference.

[user@server ~]$ ntpdate pool.ntp.org

All back to normal. For five minutes. A day later it was a couple of minutes out again.

Called the hosting company's support line. Three days of head scratching later, they say, "We'll swap the drives into another box." One day after that, they say, "It's all fixed". One more day and it's buggered again.

VPN box became ftp box, ftp box became VPN box. Almost two years later:

[user@server ~]$ date;date;date;date;date;

Thu Jun 28 13:43:35 BST 2012

Thu Jun 28 13:50:09 BST 2012

Thu Jun 28 13:43:35 BST 2012

Thu Jun 28 13:50:09 BST 2012

Thu Jun 28 13:43:35 BST 2012

Smoking boots and server outages

I was once paged by a major client in order to deal with an 'outage' of nearly a hundred mid-to-high end UNIX servers at around 4am. A quick wander around revealed that every server had suffered a power outage that lasted nearly an hour, and nobody would tell me what the root cause was, and looked really shifty when queried about it. I started running file system checks and getting everything up and running again, and tried to find out what the story was.

Eventually I discovered why everything had dropped and why the Ops Bridge crew were looking guilty and keeping quiet: A UPS engineer had been working on the system overnight. Due to the vendor being tight-wads and cutting contractor rates to the bone, said sparky did not have a mate with him and was working alone on the systems, and the vendor did not arrange a full outage because the systems were too critical to business. The guy fried himself and was killed and cooked over a period of time. If someone had been there with him, they could have killed the power in time and he'd have lived (he was still alive when the first medical guys arrived, but they wouldn't touch him as the power was still on). There was some delay in power-down due to concerns that HA systems would be affected. Which shows you the value placed on human life by some companies.

So I fsck'ed a bunch of machines in the wake of this mess.

The kicker: The department in question had a great safety record and all of the vendor's staff got free stuff while the record was maintained. In order not to jeopardise the clean health and safety sheet and cause awkward questions, the death officially went down as a suicide.

Because obviously you don't work on a UPS alone unless you are suicidal...

Another classic...

Because I worked night-shift looking after a mainframe and a UNIX system, our glorious leader decided that our section would be ideal for offering overnight PC helldesk and tech support to every muppet on the airfield who had a PC. So my mid-shift naps were interrupted by a series of idiot-calls from the Air Force's finest on a routine basis.

Picture a winter's eve. The runway is closed to to bad weather. The snow is coming down and settling heavily. It is now 4am.

The phone rings.

His monitor doesn't work. The light on the front isn't on.

"Is it plugged in and switched on?"

*suspicious lack of any pause to check this before replying*

"Yes, of course."

I offer to wander up to his office to take a look, and he now reveals that he is sat in a building over the other side of the runway. It's either a ten minute walk in the snow, or a nine-minute job of defrosting a vehicle and two minutes of driving over there.

Great.

"Are you really sure that it's plugged in and switched on"

"YES!"

"Ok then. I'll be over in fifteen minutes. I'm going to bring a screwdriver and a hammer. If it's broken, I will use the screwdriver to fix it. If it's not switched on at the wall, I will use the hammer to hit you. Now: Is... it... switched... on... and... plugged... in?"

*actual sounds of rummaging*

"Oh... it's working all of a sudden..."

*hangs up*

I used to hate handing out root passwords or any kind of admin rights to users, regardless of the number of letters from their boss telling me that the said user 'needed' it. I cannot count the number of times I've had to rebuild stuff that users screwed up.

The worse example was when I gave someone rights to install software on his machine. The next day, I was called to his office. He'd tried to reinstall Windows for some reason that was never clear (I suspect he'd deleted the files labelled 'Windows' because they didn't appear to do anything useful). The problem was that he couldn't fit the floppies into the drive any more.

The 'any more' should have been my warning.

A visit to the office showed me that the machine was asking for the user to 'insert disk 4'. The user had successfully inserted disks one through three when requested. Concurrently. Without ejecting prior ones. They were all jammed in the drive.

This was a guy who only 24 hours had received a written testimonial from his boss tersely telling me that he 'knew what he was doing with computers'. Clearly the guy DID know what he was doing on some kind of zen level of pure stupidity which I can never understand... because he'd actually got as far as disk 4 in the procedure successfully!

A few

Story 1: I work at a small regional bank in the US.

As is often the case in a smaller IT shop like mine things seem to either be completely dead quiet or insanely busy, often with no rhyme or reason as to why since the problems that come in are often not related. A PC goes down in one office and not a few minutes later someone else has a problem in another part of the building. After several hours of people calling in with frantic requests for help - since their problem is obviously the most important thing - it can get to even the most seasoned of sysadmins resulting in a increasingly short temper.

So near the end of a particularly rough day, I get a call from a teller who insists that they can not print out a transaction receipt for a customer to their little serial printer. I run through what I think is a pretty good checklist. Is the cable connected to the device? Is the print session to the AS400 open properly? Does it have power? I check the print device on the AS400 host and its online too. This of course all takes time to do but the employee is having none of that and is becoming increasingly agitated because of course if we don't fix it in 20 seconds then we're incompetent.

"Fine I'll be out in a minute to replace it," thinking I'll just swap the stupid thing and worry about it in the morning.

Arrive out there, and the teller and the teller's boss are fuming about this because they had to hand write the receipt, oh-heaven-forbid. I open the top of the little Epson receipt printer to swap out the paper roll and see the problem. "There's no paper on the roll." I tilt the printer so they can see, shut it, and walk out without saying another word.

Story 2:

"My keyboard won't work anymore" the person in the mail room tells me, "So I can't punch in to the time clock." There's few things that upset a user more than not being able to punch in immediately when they get to work. The idea of staying an extra minute or two at the end of the day to make up for it is often met with the shrieks and cries of the damned. Fortunately he called early in the morning so I'm in a relatively good mood and his pained cries for help are easily brushed aside.

"Okay, probably just broke down, we have plenty of spares. Maybe you could punch in from someone else's desk and I'll be right down." Go to the storage closet and grab a spare, hoping the rest of the day will be this easy.

Get there, spare keyboard in hand and disconnect his old one. As I go to throw it in the trash can next to his desk a stream of water comes out of the keys, resulting in a puddle of water on the carpet. "Um, did you spill something on this keyboard?"

"Well, yeah, I spilled some water this morning. Does that break them?"

"Well, computers run on electricity and these are electronics, so yeah."

"oh.."

Users..

Story 3:

Problem with the AS400 host system of some nature and users are in a panic. They can't process loans, can't open accounts, can't do anything. After the first few calls from users we're all getting a little bit nervous I send out an email to all users saying the AS400 is having a problem and we're working on it. The stress starts to increase when a few more issues crop up on other systems since those systems can't talk to the AS400 now either. So the stress factor starts to multiply.

So we're making calls left and right to various vendors and support personnel for assistance. While trying to do these productive things people in another department start bombarding my email with messages about how the AS400 is down. I email them back tell them we're working on it. But the emails keep coming.

After the problem subsides I go up to tell them personally and directly that everything's working now. They are all laughing and said they know they were just pulling a little joke on me.

So the next morning before they came in I went up there with some tools and pulled all but one wheel off their rolling chairs. This resulted in security being involved in trying to figure out what happened and multiple emails going around asking for information. I let those emails circulate for a while and then told security what occurred and where to find the wheels. I left them behind the desks trash can of each user. I wasn't written up and that group didn't call the help desk for about two weeks after that.

In my first job technically before I became a sysadmin, I worked at a top UK university. One day a female computer science professor came in holding a rather smelly and damp-looking Apple keyboard. 'It's broken' she said. 'How' I asked. 'Well, I threw up on it this morning and I figured the acid might damage it so I hastily tried to clean it off' she replied. Okay... 'What did you use to clean it with?' I asked thinking she'd dunked it in water or something marginally less stupid. 'Cooking oil', she replied. 'Hmmm, maybe we need to get you a new one then...'.

Don't you love academics? I had a few people wash laptop keyboards under a tap as well, with the laptop still attached of course.

competition is only for US citizens

which is a shame, as I can sympathise with your mate Bob

No RAID

This one almost sounds like an urban legend. I wouldn’t believe it myself, had I not run into the handy work of the ‘IT Guy’ in question myself. It happened before I started working where I am now, so isn’t my own story.

When taking over a large amount of outsourced IT work from another company, it was discovered that some of the servers our company were now looking after happened to be running on single disc drives. When the previous ‘IT guy’ was questioned about this lack of redundancy he absolutely insisted he’d done his homework and fitted mirrored discs. He even went to the trouble of opening up one of the systems in question and pointing to the polished (as opposed to brushed or painted) finish of one of these ‘mirrored’ drives......

Many years ago, I worked in 3rd Line Support for a reseller. We had internal IT for our internal systems, but as usual, when there's a problem, "you're a techie - fix it" is usually the way forward.

Arrived at work before 9 o'clock one Monday morning to be greeted by the Sales Director to inform me that no-one could access their email. Popped into the server oom, and found that an entire rack was down. I quickly disvovered a faulty power strip, and as there was already a spare the other side, began moving all plus across to it.

I had to connect the new power strip under the floor of the server room, so called one of the junior members to help with the lifting of tiles, and to make sure no-one fell down the hole when the floor was up. It all went well, and a short time later the rack was booting up nicely, so set off to leave the server room with the junior member to inform everyone that all would be well in the next few minutes.

Now, to leave the server room you had to hit the door release button next to the door. Unbeknown to the junior guy, next to it - under a cover that had to be lifted - was also an emergency shutdown panic button that would kill power to the whole room. Yep, you guessed it - he lifted the cover and hit what he thought was the door release. 20 server racks powering off left the room feeling ever so silent!

Cue spending the next three hours getting all of the systems up from scratch. :(

General hospital

I once had to spend a weekend installing servers at a hospital. The on-site tech told me the story of their nice new shiny server room.

They got all the servers in racks, rolled them in, and went to turn them on. Hmm, something not quite right here - no-one's installed the power yet!

OK, roll them all out, install power. Roll them in.

Hmm, no network! Bugger. Roll them out, install network, back in.

Hmm, it's getting rather warm in here. Ah, maybe we need air-conditioning?

A few weeks later, air conditioning installed. Now we're getting somewhere.

Except the air-con wasn't installed properly and flooded the server room several feet deep in water.

Roll the servers out, dry them off, install a drain in the floor just in case, roll them back in. Yes, the dried-off servers still worked. They're braver than I am re-installing flood-damaged servers but that's maybe that's the way they do it in the NHS?

Oh the joys of good server room design!