Feeds

back to article Sysadmins: Your best tale of woe wins a PRIZE

I'd like to share with you the plight of a good friend of mine. He's a systems administrator for a mid-sized American accounting firm. His story isn't particularly remarkable, but all the more important because of it. Bob has a bachelor's in computer science, and a lot more experience than I'd normally expect someone clutching …

COMMENTS

This topic is closed for new posts.

Page:

Thumb Up

This sounds like a way of soliciting new stories for the Bastard Operator from Hell series!

2
0
Bronze badge
Windows

Well not strictly true

Like many of us I take the lessons of the BOFH and the PFY to basically be a git.

0
0
Anonymous Coward

*rubs hands*

this should be good, El Reg readers always seem to have a never-ending stock of stories that amuse, entertain and horrify in equal measure

2
0
Happy

RBS?

Any candidates from RBS? :-)

4
0

Re: RBS?

This is their new recruiting drive, they are waiting to head hunt someone with a story about CM 7

0
0
Silver badge

Re: RBS?

CA-7 even? Or were the BOFHs playing Championship Manager when they should have been working?

3
0

Re: RBS?

Lol,

They were probably to busy hacking it and pirating it for later.

I must of been playing on something or slacking to not spot my mistake there....

0
0

One off the top of my head....

A foreign system that I managed was down for a lot longer, than it should have been, just because I couldn't find the key in the remote session to produce the "£" symbol in my password. Always use "$" instead these days.

7
0
Anonymous Coward

Re: One off the top of my head....

This did make me laugh, cheers!

I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware), working out which one has caps lock or sticky keys switched on can be a pain!

3
0
Bronze badge

Re: One off the top of my head....

@moonface

There are times that I hate those who mandate "special characters" in passwords.

I'll hazard not one of them has had to use the wide variety of foreign language keyboards that I do.

And don't even mention "dead key" keyboards to me...

And that's assuming that the login prompt is configured for the same keyboard layout as the user session.

Mine's the bag with the US keyboard in it.

0
0
Bronze badge

Re: One off the top of my head....

"I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware)"...

I've got a mixed language combination of RDP bounces which won't let me get at things like ~, [, ], \.

And folks wonder why my command prompt has these weird characters in it :-)

1
0
Bronze badge
Facepalm

Re: One off the top of my head....

UK Apple keyboards don't seem to have a key for #, which makes logging in difficult if you have one in your password.

And speaking of non English keyboards, Turkish keyboards have a selection of letters which look like, but aren't, English characters, which also makes typing in passwords difficult, should one be trying to check ones gmail from a hotel.

1
0

Re: One off the top of my head....

Sounds very familiar to the time I had remoted onto a virtual session hosted in India. Had the user on the phone and was desperatly trying to find the local keyboard setting for ". I was unable to and then spent a good 10 minutes trying to convince the user that all I wanted was for them to type " into a command prompt. In the end I email the " to them and copy/pasted it in.

3
0
WTF?

Re: One off the top of my head....

It is Alt-3 which gives #

0
1

Re: One off the top of my head....

Option-3 gives #. Keyboard viewer is your friend!

1
0
Silver badge

Re: One off the top of my head....

Was your password for dial-in on-call fixes "£4nothingchicks4-3"?

2
1
Trollface

Re: One off the top of my head....

No, it was "Sh!tSyst£mFu(K£d"

now it's "$h!t$ystemFu(Ked"

1
0
Bronze badge

Re: "Keyboard Viewer is your friend"

Yes indeed, but sometimes you cannot get the right characters at a Terminal prompt.

The answer here is to add US Keyboard layout to your options in International. You can set up a shortcut such as Cmd-space to toggle input languages..

0
0
Silver badge

Re: One off the top of my head....

You can get it through a key-combination (right-alt 3, I think? I can't remember without actually being sat at the keyboard), but this is rather non-obvious. No user would guess that, I had to look it up. It also doesn't work when I'm running windows and RDPed into another windows server - whenever I want to join a channel on IRC, I have to copy-paste a hash from somewhere.

0
0
Silver badge

Muppet engineer

We once purchased a server with Windows Server pre-installed. By the time it was delivered plans had changed and it was reformatted to run Linux. Three months later it broke down. Looked like a simple failed PSU to me, but it was still on warranty, so we called for an engineer.

Some hours later he told us it was OK again and left at a trot. We were surprised that he hadn't left it powered up, and dispatched someone to the machine room to boot it. But it was booted ...

and once again running Windows! The muppet thought GRUB was a hardware error, so he reformatted the disk array and reinstalled Windows. Thank heaven for backups, and that it wasn't desperately mission-critical.

You don't have to outsource to India to get muppets.

11
0
Bronze badge

Re: Muppet engineer..thought GRUB was a hardware error,

Would have been shot for reformatting ANY of our servers with WindblowZE.

We don't have these nice Tux stickers and posters around here for our amusement. The door to the server room doesn't say "Penguin Lounge" for nothing!

1
2

Slight issue

If you go there and spend ages typing something in then you will lose all the text unless an already logged in member of their community. To register they seem to want a huge amount of your details before you can continue.

0
0

Re: Slight issue

Probably less of an issue than you might think, since most readers seem to have completely missed the point and are busy typing their stories into the Reg's comment form instead.

1
0
Silver badge

Mouse problem

I was called in to fix a workstation in an old Victorian basement bit of the site. Quad 140W Opteron thingy. Expected to have a certain amount of fun sourcing a beefy enough power supply. But the lights were on and fans were whirring.

When I took the cover off, a very strange burned organic smell assulted my nostrils. A few seconds later I found a dead mouse with its head wedged between the fan blades and one of the heatsinks. I hope the poor wee thing's neck was broken in an instant. I fear otherwise.

After the mouse was removed and the CPU allowed to cool down, it rebooted without a hitch. The mystyery remains, how did a mouse get into the case? There was no hole anywhere near big enough anywhere in the metalwork that I could see.

2
0

Re: Mouse problem

Mice can squeeze though gaps as small as 5mm so looking for a mouse sized hole means you're looking for a tiny little hole.

0
1
Silver badge

Re: Mouse problem

It slipped in as a mouse kid and fed on whatever came crawling along... Poor thingy. Presumably it committed suicide to end its meaningless life - I mean, who in his/her/its right mind would want to spend the life poking in computers?

9
0

Re: Mouse problem

The rule when dealing with mice is that anything you can stick a pencil through isn't mouse-proof.

I never found out if that referred to the diameter of the hole, or whether a sharpened pencil could be shoved through.

Probably either.

1
0

Re: Mouse problem

What a great story. I work for SolarWinds and would love you to enter our contest, but unfortunately, we are not yet able to open our contests up internationally. We're trying to work that out with legal and hopefully will be able to offer our contests to all soon. For those of you who do live in the U.S. and would like to enter, here is that link: http://thwack.solarwinds.com/community/solarwinds-community/sysadmin-day-contest.

1
1

This post has been deleted by its author

Gold badge
Happy

Re: Mouse problem

"If you get a pencil through a hole a mouse can !"

How does the mouse hold a pencil?

1
0

Re: Mouse problem

Dunno who downthumbed you. I did pop by and submit a story, and got some rather nice friendly PM's regarding it - seems like a nice community over there. Not being able to win the competition is kind of a bonus anyway - the price is something Apple made called an "iPad" or something. Pretty sure I wouldn't know how to use it.

0
0

I reckon the boys at RBS will have this competition in the bag

1
0

"I reckon the boys at RBS will have this competition in the bag"

They did but unfortunately they left the bag under a leaky sewage pipe and then threw it out when they noticed it smelled funny.

0
0
Joke

Didn't they out-source the bag to India?

5
1

Not the bag

Just the guy who was supposed to be carrying it. Turns out they didn't bother to check if he'd ever seen a shoulder strap before...

1
0
Anonymous Coward

Too many to choose from...

Starting with the MD who thought he was still up to speed technically speaking and factory defaulted all of our coms gear to the helldesk tool who decided the best way to free up some space on a development server was to delete the contents of the system32 folder!

0
0
FAIL

Re: Too many to choose from...

Years back, we had a 'Unix consultant' who decided that he'd empty the user trash can on the server - the directory called /usr/bin - then wondered why the server stopped working.

10
0
Silver badge

Re: Too many to choose from...

One of my ex's did that to her laptop.

Luckily, she was hot and I forgave her.

I may also have hammed up the difficulty of fixing it a bit, too.

2
1
Facepalm

I'm allegedly a developer...

I'm allegedly a developer... (admittedly one with a CCNA and an interest in system administration, Linux and other general geeky stuff)

Except i am charged with Running exchange, the cloud platform, the virtulisation stack, all of the servers and desktops, the helpdesks (yes 2, as a so say manager was beguiled by a nice mug and stickers, didn't even hold out for a pen drive!) , documentation, router kicker (crappy dlink wireless router from PC world powers the office, and dont get me started on the ADSL line!!!!), 1st, 2nd and 3rd line support, explaining basic concepts to the "technical" director (yesterday talking to a potential client his response was "we send some HTTP", when asked a question about SSL) oh and taking the flack for when dev deadlines are missed for all of the above, well here are some WTF's from this job:

1) We upgraded to SBS 2011 becasue the MD bought a mac book and couldn't connect to exchange, said mac now lives on a shelf gathering dust, Migration from SBS 2003 disnt work and we spent 3 weeks with a partial service.

2) Exchange is hosted locally on an ADSL line which has a top down speed of 5Mbs, on a router which crashes 10 to 20 times a day, average mailbox size is 12Gb and the suggestion of archiving went down like a lead balloon and was forced to revery back to all you can eat inboxes.

3) one of the helpdesk servers was a windows XP box bought in 2002 which died last week, that was

fun to fix (now on a proper server)!

4) we have a mixture of 1 to 10 Gig ethernet to every desk yet the MD insists on people using wireless to said crappy router because wires make the place look untidy...

5) It took 6 months of nagging to take the UPS out of the empty comms rack and install it in the server rack

6) The working helpdesk has been replaced by a cgi based web app which crashes when more than 5 people try and access it it also uses a VBScript based file upload and a scheduled task inplace of a service and requires administrative permissions to save to the file server, suffice to say new clients dont have the option to add attachments to support calls (we publicised the "upgrade" to our clients most are not impressed!)

7) A mezzanine floor was removed, but a comms cupboard was left 20' up a wall, i needed to replace a switch in there, new switch wouldn't slide in shoved it harder heard a crunch then a loud buzz, there was a bloody wasps nest in the cupboard!!!! queue rapid decent down the ladder and no network to half the building until the exterminators had been!

I could go on but the phone is ringing and someone needs their hand holding...

I Love deadlines its the whoosing sound they make when they fly past....

7
0
Anonymous Coward

Re: I'm allegedly a developer...

Are you me from the future?

our sbs2003 -2011 migration hasn't happened yet, but the cupboard and wasps nest thing sounds familiar.

Just got back from holiday and found:

one of the HD in our RAID array has failed and the guy called in just took the faulty disk out without replacing it.

The cooling to our server room was switched off because it was too noisy

the "internet has been slow" so someone has been hard rebooting the servers every day by unplugging them.

My work area has been "tidied" by putting all of my components and tools onto the warehouse racking at ceiling height.

0
0

Oh God

You tried to migrate straight from SBS2003 to SBS2011? You poor, poor bastard. I spent two weeks once on a 2003-2008 migration -- eight of those ten days with MS Product Support on the phone eight hours a day, for which service I ended up paying not one thin dime, and if you know how MS PS support works then that tells you all you need to know. (If you don't: Unless you have a support contract, which costs $stupid, there's a per-call incident fee of merely $silly, which is waived if the cause of the problem turns out to be Microsoft's fault. They waived it.) And I've got a 2008-2011 migration coming up this weekend which I am sure will also be a nightmare. Straight from 2003 to 2011? I'd rather have a root canal.

I could go on too, but there's a server to bring back up and a couple things to set live before the day's work proper starts off -- back to the coal face...

0
0
Facepalm

Re: Oh God

It wasn't fun i can tell you that, and i have had a root canal before, was definitely more enjoyable, lots of painkillers straight on the nerve :D..., 2008 to 2011 is meant to be less painful but still a royal pain...

Best bit is the one unfixable problem is that the one person in the company who needs access to every ones calendar is the one person who's account wont allow the permissions to be set oh the joy, tried explaining to "management" that exchange nevermind system administration is a full time job in its own right oh well u can but try...

0
0
Silver badge

Simple things

A PC wasn't working, looked like PSU fault. Took it back to my office to check and repair. Plugged in. Pressed power button. Very loud bang. Small amount of smoke. Lights went out. Oops.

Outside office, also no lights. OOPS.

100 yards down corridor, still no lights, lots of people peering out of their offices asking WTF. OOOOOPS.

Somehow, the PSU fault hadn't taken out the 13A fuse in the plug (yes, it should have been a 5A fuse, but it did have the mandatory safety-tested passed-by sticker). Not the 30A breaker on the circuit either. Nor the 100A breaker above that. No, when the electricians finally located the fault, it was a 180A fuse in a box high up on a wall that had probably last been looked at when it was installed in the 1920s.

It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds. As far as I know it's still there. Too much effort to schedule replacing it with something modern before the next time (in the 2060s? )

The PC was fine after a PSU transplant.

You don't need complicated things to make a FUBAR. The simple things are also out to get you.

I could also tell you about the exploding substation and the need do do a tap-dance to avoid getting burned by globules of molten copper pouring out under the door, but that's got no IT angle at all.

8
0
Silver badge
Coat

Re: Simple things

"it was a 180A fuse in a box high up on a wall"

That's a 6 inch nail isn't it? Or would be in the BOFH stories...

"It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds"

Take a note of your nearest B&Q for the next such outage :-)

1
0

Re: Simple things

Having worked in B&Q many moons ago, they wouldn't know a 180 amp fuse if you rammed it where the sun don't shine. As for them stocking one, you have more chance of the devil calling a central heating repair service!

0
0
Gold badge

Re: Simple things

"As for them stocking one, you have more chance of the devil calling a central heating repair service!"

You've given the answer yourself: they DO stock copper pipe.. :)

1
0
Anonymous Coward

Re: That's a 6 inch nail isn't it?

Consumer unit fuses are bigger than the little cartridges in 13A plugs, and big industrial ones have lugs on the ends - a bit of copper pipe of the right size should do the job. :-)

0
0
Silver badge
Facepalm

@Callam McMillan

Hmm... B&Q do still stock 6" nails though, right?...

0
0

Re: That's a 6 inch nail isn't it?

When I was a callow youth I worked as junior shit-shoveler at a medium sized engineering firm. One afternoon there was an almighty bang and the whole site went dark except for the emergency lighting. It seems both the main HT connector fuses had blown when one of the company sparkies grounded something he shouldn't have (and was very lucky to be alive). Anyway, need new HT link. Head sparky starts making calls but none to be found this side of the country. Nearest replacement is 1000km and two days away. No mobiles back then, so all this had to be done in full public view in the front office, with a growing and restless audience. Management by now is soiling it's collective self about downtime and missed deadlines.

Cue junior sparky and a "hang on lads, I've got an idea" moment: one piece of one inch copper pipe, two blows with a hammer to flatten the ends, punch two holes to fit over the terminals and voila! We are back in business...

2
0

Bit of a WTF situation just now

10 months down the track we are still trying to give us money to replace our primary (read only) flile server. If it powers down it will not come back until the following routine is followed:

Unplug cable, pop lid, remove BIOS battery, bridge reset, insert power cord, remove cord, unbridge, replace battery, close lid, power up, reconfigure the BIOS again.

Combined with the fact that we are still trying to also get money to replace the faulty tape drive. It goes for a couple of days then fails with unknown HW error. Only solution is to power down drive and server, power up drive, wait, power up server. Sometimes it works first time, sometimes takes a few tries. Our last successful backup was 9 days ago.

1
0

Page:

This topic is closed for new posts.