back to article A sysadmin's top ten tales of woe

Get enough people of any profession in one room and the conversation drifts inexorably towards horror stories. Everyone loves a good “…and then it all went horribly sideways” yarn, and we all have more than one. The IT profession is flush with tales of woe. There are no upper boundaries on ignorance or inability. From common …

COMMENTS

This topic is closed for new posts.

Page:

Anonymous Coward

Heard of so many....

1) Somebody plugging in a KVM to a solaris server, but only the keyboard worked. Go to KVM to use a different machine, see the black screen and assume windows is crashed -> CTRL-ALT-DEL -> Critical Solaris server reboots.

2) Calling BT out for a failed NTE after a power down. Only when they arrive we realise it was plugged into a different power bank which wasn't switched on.

3) Customer required some security software on all machines to prevent unauthorised software and hardware running. It worked off whitelists provided by a seriously overloaded server. Despite warnings the customer wouldn't upgrade it. One day the server throws a hissy fit and provides corrupted whitelists which blocked NT.DLL, preventing login on all machines which had updated the whitelist.....

0
0

Staff Efficiency

Many years ago in a universe far far away, I was the IT managing partner in a small architectural firm. We used a Wang minicomputer (remember those?) for bookkeeping duties that was generally managed and maintained by an outside service.

One day the service technician walked into my office with a face the color of fresh clean photocopier paper. He explained that he had made an error in upgrading some routine and the Wang was not working. He mounted the supposedly current backup (the techs always called ahead and asked for the staff to do a fresh backup before they came over to make changes) however the newest backup was showing 8 week old data. These backups were done on 8 inch floppy disks that were verified at the end of each set. We had 4 disk sets in rotation, all properly numbered, verified and dated. Sure enough, all the current sets including the one dated earlier in the day had 8-week old data.

So I asked the bookkeeper to show us how she had done the backup. Instead of going to the regular backup screen, she pressed a combination of keys which flashed a warning signal, "For computer technician use ONLY. Bookkeeping personnel DO NOT USE." Near the center of the screen was an option titled "Add updates to current backup." The bookkeeper said, "I've been using this update for the past couple of months. It's a lot faster than the regular one. I don't know why they don't want us to use it."

The technician turned even whiter, checked the backups again and sure enough. The backup record showed that it had been run daily for the past two months, dutifully noting that there had been no system updates during that time. The technician finally asked the bookkeeper to do paper printouts of all the reports on the system, which the bookkeeper grudgingly agreed to do. Later that day, the technician walked out with two boxes of printout and the hard drive from the system. We hired some help and switched to manual bookkeeping (payroll that day as I recall).

A week later the service boss called and said they were unable to reconstruct the data from the hard drive in a meaningful way. He offered to loan us an identical computer so we could reconstruct the system by entering the data by hand from the paper reports and proposed that we share the cost. I finally convinced my senior partners that this was a fair arrangement. The service also modified the Wang dos so that updating the system would never overwrite valid data. We were back in operation in about 3 weeks.

We eventually outgrew the Wang and switched to a miniPDP, but that's another near-tragedy for another time.

John

0
0

Power failure

Power failure in the building, go up to the sales department and find out that none of the sales staff computers are connected to the UPS backed sockets but the Christmas tree is: nice blinking lights!

4
0
FAIL

Or..

Who bought a bloody great UPS... with no batteries in the early 80's and never tested it.

Then one day.....

0
0
Thumb Up

Re: Make sure the backup is going where you think it is going.

Put scripting in to estimate how much data you're going to backup and then query the backup solution to find out how much (file count/overall size) it actually backed up...

Then get that pumping into a DB with alerting if it backs something not consistent with history.

Oh and do restores - just to check, they're more important than the backup :)

2
0
Vic

Testing your Makefiles...

This one happened just a few weeks ago.

I was on-site at a customer. I accidentally overheard a conversation.

The group in question had a makefile which would take a project name and apply the make to that - so you'd do something along the lines of "make PROJ=foo all". In this situation, the user had typed "make PROJ=foo clean".

Except he hadn't.

He'd typed "make PROJ= foo clean". And that spurious space meant that PROJ was defined as null within the makefile.

Did the makefile do something sensible with null values? Did it buggery. It merrily deleted everything from the current directory down - including all the user's source code. Which wasn't checked in. Or backed up. And hadn't been for nearly a fortnight.

I earnt many brownie points for pulling out a copy of foremost :-)

Vic.

0
0
Mushroom

This is starting to sound familiar...

The first time our computer room UPS failed to operate during a power cut, we called out the service engineer who dutifully arrived on site and pressed then "ON" button... Oops. It had been sitting there for nearly a year doing absolutely nothing !

The second time it failed when one of the large capacitors on the rectifier went bang - quite literally. I still remember vividly, talking to the UPS engineer over the phone and asking him to confirm, again, that it would be ok to flick the breakers to put the thing into bypass mode, whilst smoke was still pouring out from the top...

0
0
Happy

/me sniggers at jwkinneyjr

regarding Wangs not working properly and being too small.

0
0
Mushroom

Not the cleaner for once.

Way way back at the Uni we had a cleaner unplugging stuff on a regular basis.

So one day when a server was down i made my way over to the dingy "server room" to plug it back in.

Opened the door and saw a rack with halfmelted servers slammed the door and called the fire department.

Turned out a holding tank of sulfuric acid on the floor above had sprung a leak.

1
0
Facepalm

Virtualization blues

Good idea: converting a dozen separate servers to virtual containers on OpenVZ, freeing up a whole rack.

Better Idea: having a second OpenVZ that mounts/runs the containers from the shared NAS so you can shove containers from one to the other in seconds.

Bad idea: running debian updates on the primary OpenVZ host without testing them on the secondary first.

Result: server finds no OS at next reboot. Cue failover of critical containers to secondary, (yay!) and all-nighter trying to fix primary (woe!), with every step seeming to just make things worse, until it ends in a clean reinstall of the OpenVZ host.

Cause? Grub update silently changed the device.map

Cunning idea: writing a script that will run the same command on all the virtual containers you list on the command line.

One careless moment later: renaming the script to 'clusterfuck.sh' so anyone trying to use it knows what they're risking.

Also: NAS (Qnap in this case) firmware updates are evil.

1
0

yup

Tales from the Crypt of 15 years in the IT world.

1. A certain global company had 3 operations centres. I worked in the EMEA one. The server room had lovely UPS rated for a couple of hours and was hooked up to an external generator.

When the local utility company sawed through the mains for a laugh, everything went A-OK. When it became obvious the generator would be needed, it was spooled up, but it required someone to be physically in the server room to let the juice from the generator in (a security feature apparently). Except, the only way into the server room was via a badge reader. Yup, the lock was electric and not hooked to the UPS. The only key holder for the physical lock was 10 hours flight away..with the key. Queue panic and IT managers running for the fire axes.

2. Same global company later outsourced half the ops work to a famous Indian outsource brigade. The lovely IT campus on the East coast of the subcontinent had survived the tsunami a few months earlier, but the whole area was regularly flooded. I was there during one such flood. The servers were safely tucked away at least 5m above the highest theoretical tsunami/flood mark. Except the emergency generators were....in the basement! Queue a small army of locals shifting sandbags and running bilge pumps/chucking buckets by hand.

3.Same company had operations in just about every landmass with more than a couple of thousand inhabitants. The SAP rollout required as much data as could be sucked in - meaning IT infra in places served by meagre comms - and lacking in anyone vaguely qualified in IT. The best they could be expected to do was wire a plug. One local expert did well to physically build the server and attach the storage. It had been shipped imaged - already in the domain etc - then stripped down for the journey. The build process required a testing of the RAID array, which is where it all went horribly wrong. Queue 3 day journey for the 3rd level support guru by train/plane/boat to discover the poor local had thought that testing the hot-swap meant taking out all the disks sequentially (at the same time) and putting them back.

4. A certain behemoth of an IT company I worked for. We needed to move an NT server that had become mission critical from an office space to the nuke-proof data centre at the end of the corridor, it genuinely did have real 1960s blast doors, sprung floors, faraday cage etc). Months of planning, contingencies coming out our @rses. One thing we didn't think of was that the machine hadn't been cold booted in about 4 years. Click, click, click. Oh dear....processor dead. In those days you couldn't just whack the disks in another box, it had to be exactly the same hardware.

I could go on and on....

2
0
Silver badge
Coffee/keyboard

Classic

"Yup, the lock was electric and not hooked to the UPS. The only key holder for the physical lock was 10 hours flight away..with the key. Queue panic and IT managers running for the fire axes."

Running for the fire axes? You owe me a new keyboard - excellent story.

1
0
Flame

Air con woes

And in the same room as the exploding UPS we had some spectacular air conditioning failures...

Most of these could be attributed to the senior management who insisted on going with the cheapest quote when upgrading the air con, to cope with increase in the number of servers.

Despite being told, time and time again that you need proper computer room air con, they insisted on going with the cheapest office comfort cooling units they could find.

Some of the highlights..

1. The external air con units vented out into the warehouse rather than the open air. They quite regularly got turned off by the night shift in the warehouse because they didn't like having warm air being blown onto them - until we put padlocks on the isolators !

2. One of the "new" air con units to increase the cooling capacity had a faulty condensate pump. The resulting "overflow" created rather a large puddle under the floor, where all the power outlets happened to be. Fortunately it didn't get too deep to trip anything out !

3. Air con units designed for office environments don't like being run at full pelt 24 x 7, and they tend to shut themselves down when they get too hot....(which somewhat defeats the point of them being there) The ensuing "melt down" as all six units decided that they'd got too hot to function was quite interesting to say the least. I still remember nearly burning my hand on one of the server racks ! It took nearly a day to get some of the servers cool enough to work again.

Needless to say the "I told you so" fell on deaf ears.

1
0
Silver badge

The Snow bomb

Great story on the daily WTF about a US university that cut into an unused elevator shaft to lower some big iron into the basement machine room. They then covered over the shaft at the bottom - not the top.

After a winter of heavy snowfall came the spring thaw - and a 5 storey plug of snow and ice melted into the machine room. Quite impressive tsunami apparently.

1
0
FAIL

RDP chain

Note to self - When RDPing from one server to another, check, double check and triple check which machine you're on before clicking Shutdown.

I thought I was on Cape Town's backup server, evedently it was Frankfurts prodution one...

1
0
Anonymous Coward

Brand new access card system...

... all controlled by a server behind a secured door.

You guessed it, the server failed. The on-site fire fighters soon chopped through the door with an axe.

0
0
Anonymous Coward

Do talk things through with colleagues, don't blindly trust automated protection

In the past I managed, with the assistance of a friendly UNIX admin, to set up a mail loop between an Exchange system and the OfficePower setup it was replacing. Each of us had taken it upon ourselves to set up mail synch, pushing new mails out to the other system - both worked very well.

Well, up to a certain volume of mail.

We were on our fourth lunchtime pint when someone arrived to drag us out of the Cross Keys.

-

On a similar theme, the same organisation trusted mail loop detection enough to allow some senior staff to set up automatic forwarding to external mail services.

Sadly a "This mailbox is full" auto-response from the automated remote system admin was considered a new mail and not part of a loop so when the then-stingy webmail box was full it merrily informed the sending system of the fact which dutifully forwarded the information on . . .

0
0

I've seen a few of these as well

Every 3rd business day at 1 of the ISDN sites for a company I used to work for, would lose all customer connectivity.. turns out the the cleaning crew had access to the comms room and would unplug the Remote Access Server to plug in the vacuum. Why you need a vacuum instead of a duster is beyond me???

Also every friday at 3pm for the same company in the building we had just built, customer connection service levels would plummet but network traffic would soar... Seems there was a large group of people who would fire up nice rousing game of Duke Nukem and kill the network.

1
0
Facepalm

So you're root? Unfortunately, yes.

Let's make a small contest here, what went wrong:

I happened to be an outsourcing slave for a while. Job indluded to fix everything, what the customer, HAVING ROOT AACCESS, did to the system(s).

Once a while, one system shows really strange and stupid behaviour. On a Sunday, naturally. Buddy calls me if I can take a look. No network access. Drive to the office to look at the serial console. Ok, looks like the whole OS is gone ?!?!? Must be the bootdisk, then. Not mirrored, 'cause a 2nd disk costs real money, doesn't it? Ok, fix the OS, thanks god, SAP-Oracle database is till there. Next day, same issue. Machine looks and feels strange. Os is gone again. Fix that da*** disk and call the bloody hardware vendors support. They swear that the disk is physically fine. Next day, you guess it, Os is gone again. Fix/restore from backup.On the 4th day, same problem, I do a fresh install of the OS to a brand new disk. From now on, system runs like a charm. "See, it must have been the disk!!!" No.

2 weeks later a colleague gives me a call: "Hey, problems on that system start over again, OS is gone!"

Me: "What changed???"

Him: "Mr. <customer root user> asked me to enable his shell-script via cron again."

Me: "So let's take a closer look at that script now..."

#!/bin/sh

cd /<somedirectory>

find . -mtime +30 -exec rm {} \;

#EOF

Hint 1: What happens, when /<somedirectory> doesn't exist ???

Hint 2: What's (quite often) the home directory of root?

0
0
FAIL

root

one time (and never again) I let myself be persuaded to give root to a developer on a solaris 8. next thing you know, really weird fails: you can ssh in but a lot of the system commands throw errors or just fail outright, not all though. some digging revealed that everything under /usr/lib was gone. Solaris 8 system utils were mostly dynamically linked (which is a design fail in itself - try to fschk /usr after a power fail and you'll see why).

0
0
Vic

Re: root

> I let myself be persuaded to give root to a developer

I once had a wonderful argument with a customer.

He owned the box. I commissioned it. He *insisted* on having the root password.

I took the password to the office in a sealed envelope and got them to put it in the fire safe. I told them in no uncertain terms that if I saw a login from any address but mine, there would be no warranty whatsoever, and all repairs would be chargeable at full cost.

Said punter was apoplectic, claiming that he should have the ability to do whatever he liked with his own machine. "You do", I replied, "but I'm not responsible for picking up the pieces". The rest of the staff generally decided to stick to my recommended way of working, and the root password has still not been touched.

...Which is good. A few weeks after this bust-up, I installed Mediawiki for them. The same guy had another fit when I refused, without written authorisation, to configure it to allow PHP script on the pages to be executed... :-)

Vic.

2
0

Before you go, can you just...

One of the IT systems admins got asked to do a 'quick' job on a day when he really, really needed to leave on time.

Five minutes after he had to go the job was done.

shutdown -h now

"That's strange. My system doesn't seem to want to shut down." he says, shortly before all the phones start ringing.

"Remote connection closed" appears on the screen...

0
0

sometimes it takes a little detective work

Some of the stories here remind me of my first job as a sysadmin, back in the days of very low density removable disk pack drives. Space was so tight that most of my time was spent trying to find stuff to delete. Once I figured I could save some space in the root partition by stripping the kernel of its symbol table. That was a long night.

In this same job, once a month, I'd come in Thursday morning and see a bunch of new disk errors. Head crashes happened now and then (disks weren't very reliable back then) but the fact that the great majority of them happened on a Wednesday was bizarre.

So one Wednesday I camped out in the computer room. Just before midnight, a pair of janitors came in with a floor buffing machine and proceeded to buff the raised floor of the computer room, banging THUMP.... THUMP... THUMP into the disk racks as they worked down the rows. I chased them out, and next day met with the building manager, and COULD NOT make him understand that (a) the raised floor did not need to be BUFFED, and (b) the act of slamming the buffing machine into the disk racks was causing thousands of dollars in damage.

Finally I locked out the janitors by changing the combination on the push button door lock. Since I wasn't given the "reset" key to this lock, I had to take it apart, figure out how it worked, and change the combination with a screwdriver. The building manager complained to my boss that the janitors couldn't get into the computer room. My boss told him to get stuffed.

What I did not understand at the time was that the floor *did* need to be dusted occasionally. Then one day the halon alarm went off, fortunately when one of my operators was in the computer room. I ran downstairs and found him looking panicked as he held down the "abort" button. Dust had gotten into the smoke detectors under the floor. We were much more tidy after that.

1
0
Silver badge
Coat

When setting up BGP for an ISP

Always remember: Copy Start Run

That is all.

0
0
Facepalm

Ah, the memories

Over 15 years ago but.....being written up for "refusing to install a program properly" because they thought a Windows 95 program should be able to run on Windows 3.1, even when I ran the installer to show them it didn't work. Being yelled at when installing a retail copy of Office 95 because they saw the ads for other Microsoft products flashing across the screen and thought they were being installed also. Lastly, letting someone's teenage son play on my computer all day and then accusing me of looking at porn when I wasn't even there.

2
0

don't step on the orange cables...

Years ago I shared responsibility to build up a huge (for the time) decision support system. System and storage interconnect was via many strands of fiber optic cable, which for some reason was run along the floor behind the cabinets.

One morning we came in and the system was totally hosed. Eventually found out that some minor wiring repair had been done the night before, and the electricians had walked behind the cabinets in their big old boots right on the orange fiber optic cables.

We eventually got everything working again, but for reasons I can't recall, the cables still weren't put in trays, educating the electricians was deemed prevention enough, and two months later it happened again! Different electricians, of course.

1
0
FAIL

On Error Resume Panic

During my stint managing the desktop park for a household name (around 60k XP boxes in Europe, 200k worldwide...) me and my team managed to shoot down a fair few kamikaze runs. Nothing was allowed to be distributed without going through us first, and we did a pretty good job.

One minor Achilles Heel was that the high gods in the Engineering team could (but shouldn't) change the login script unannounced. Sure enough, one morning I came into a sea of white faces. The incident management team were getting hundreds of calls a minute. Everyone had a VBS error on their screen, which couldn't be cancelled and blocked the login dialog.

4 simple little words had been commented out on part of the script "ON ERROR RESUME NEXT". A bad reference to a share had caused the untrapped script to throw it's toys out the pram, and meant millions of dollars of downtime.

More recently, a heavyweight banking client of mine had an unplanned data center swap during trading hours - causing billions of dollars of transactions to be frozen (~2 hours from event to complete restore of service in another DC) . The CEO was called to the Regulators to explain how such a thing happened, and what was put in place to prevent it ever happening again. Old Murphy was on fine form that day when he got one of the SAN guys to dismount the live storage rather than redundant at exactly the time the CEO was giving his presentation. Queue automated failover, and all the local bars suddenly getting a rush of mid-afternoon customers.

Funnily enough, we'd all voiced our concerns to the fancy suit wearing US IT Consultant (Mc something...) that reducing overtime costs by doing infra maintenance during business hours was a baaaad idea. Who ever listens to the people doing the job though. McSomething and the CEO are still here, but the poor tech who thought he was working on X when he was actually on Y didn't fare so well...

0
0
Flame

Can you get me five books and a fire extinguisher please?

Working at a top university library in 1997, there were OPACs (Library terminals) - 386sx 16's, 1Mb, loading a small network stack and Telnet client, These had been in place for eight or more years (new, in Uni/Library terms) and had been unmoved, and unopened for much of that time.

Dust accumulation was apparently a problem, as one self-immolated after the PSU got too hot.

0
0
Anonymous Coward

If you're going to use RAID 5 make sure you know how it works...

Joined a small company back in '04. One of my first tasks as a developer was to build out a new Exchange 2003 server (small company, if you were in IT you did everything). After much reading up on the subject I managed to deploy an Exchange 2003 server along with a new Active Directory server. I migrated the entire company's mail from Exchange 2000 to 2003 along with implementing Active Directory to support the new Exchange system. Everything ran fine for about a week (including backups) until one particularly warm weekend. Now I should mention that IT was located on the 3rd floor of a 3 floor building. This included the server rack which sat next to my desk. The building was circa 1940 and the cooling for the severs came from 2 air conditioners in the windows which were usually turned off at the end of the day. Along comes a hot August weekend and I returned to the building early Monday morning to find that the blade the Exchange instance was running on had a blinking red light. Not knowing what it meant I rebooted the server to see if I got any useful messages. During the POST I was informed that one of the drives in the RAID was bad, but, given that it was RAID 5 everything booted up and ran normally. Once my boss (the head of IT) arrived I informed him that we had a bad drive in the array and should request a new one from the vendor. He goes over to the machine, looks at it, proceeds to reboot it and goes into the RAID controller and marks the drive as normal. I'm not sure exactly what the next prompt was but whatever he selected proceeded to take the data on the re-mounted drive and dish it out across the rest of the RAID. The data, of course, was corrupt due to the failure and the entire Exchange server I had built over 3 days went to pieces. "Well" he says, "at least we have our offsite backups". So I spent the rest of the day rebuilding the Exchange server on another blade we had received for a different project. Once it was up (around 6pm) all we needed to do was restore the data from the offsite backup. Now we had restored other files from offsite and it was pretty quick but I knew that we'd be restoring multiple GB of data. My boss said it shouldn't take more than an hour or 2 since that's how long it takes to get the backups out to the offsite location and we had a fast link to them. Apparently the up link was faster than the down link because 12 hours later we finally had the backups onsite and were able to perform the restore....

0
0
Linux

anti-viruses

i havent been working in the field for too long (a whole 2 years) so i dont have too many fun stories, but i though i should share one that recently happened.

i work for a small company and we rent a server from a company on the other side of the country, we run backups on the data and download them over dsl, as the site got bigger the downloads got bigger and we were pulling 17gigs at 120kbps. management asked me to make this better, so i made a backup that tars the data and then makes a diff of that weeks data from last weeks data to be patched on the local machine. some time later the diffs stopped patching. the md5 of the tar from the last patch no longer matched the md5 from the server. after re-downloading, and confirming the md5 i think "ok..." next week it didnt patch, the md5 of the tar had changed. after some investigation we found that the anti-virus went into the tar, found some files it didnt like, and DELETED them. this changed the tar and the diff wouldnt patch.

0
0
Coffee/keyboard

Why I buy Big Brand desktops.

When I first started in IT work, fresh out of college, my employer was in the middle of a migrate to Windows 95 (ahh, happy days. Except Win95.) They had to purchase new hardware too, despite having reasonable 486 DX100/120's to run it on. Turns out they'd bought them from a UK system builder that built their own motherboards. Which had a fault. After 30 or so failures, we were informed that there were no more spares, and they weren't making any more. That's the rest of the custom-made 1-year-old machines up the spout then.

0
0

Two tales from the 1990s

Two problems, both caused by me:

1. Changed an account latency setting in Netware (to stop people being logged in when they didn't need to be on an over subscribed server). Set the wrong value and watched as Netware terminated the connections of over 100 users in just 30 seconds.

2. Was testing conditional email forward in Lotus Notes. So, unfortunately was a colleague. We set the same conditions and forwarded to each other. The system sent thousands of forwarded emails before we could brake the loop.

Ah, happy days!

0
0
Headmaster

Backup with a punch

Act Sirius PC backed up onto five and a quarter floppy disks each night.

Customer visited to find out why second weeks backup failed.

All backup disks neatly hole punched and filed in a ring binder.

0
0

A little knowledge can be a dangerous thing

Many years ago whilst at Uni I was the main sysadmin for the SU's computing society (TermiSoc for those in the know), which had three linux servers of our very own, stored in one of the Uni building's basement.

There were a few other guys who also had root access, one of whom was very interested in security and spent a lot of time attempting to hack into and then improve our systems.

Now this guy had been reading about the risks of files being owned by root and having execute permission within user accessible folders. He started searching through the filesystem, and discovered that within each users folder there was a . and .. folder with the permissions he'd been looking out for. Now while the exact details are a little fuzzy (it was at least 12 years ago) I know our ever diligent security geek decided to fix this issue. He proceeded to change the permissions on both folders to prevent executing by normal users.

Shortly afterwards he started hearing people in the lab comment that they could no longer login. Of course removing that permission prevents a user from traversing back through the folder structure, and the login process is unable to traverse to the home directory and /etc directories. The only user able to login was root, but we'd already restricted that so remote connections were only allowed by normal users, who could then su to root, so we had no remote access what so ever.

Myself and another sysadmin friend, with resident security geek in tow, had to get someone to let us into the basement so we could get console access to the machine and fix the glitch. A fun day, but I think everyone learnt a valuable lesson, and of course the story continues to be recounted occasionally to this day!

0
0
FAIL

You gave root

to someone who didn't know what . and .. are?

1
0
FAIL

On Error Resume Next

is not a substitute for proper debugging and error handling!

0
0
Facepalm

It's in the post

In the late '80s we had a 386 setup as a bridge running software off 5 1/4" floppy. This little beastie fell over requiring me to come in an hour early every morning to download the previous day's db transactions to a 3 1/2" floppy then uploading them to the account department's system (I offered to reconfigure and bring everything up to date, but too expensive for the bean counters). 5 days of this and tracking down the guy who wrote the bridge, (he was way in the outbacks of Australia) and he agreed to send over an updated version. A few days later I get an envelope which had been folded in half somewhere on it's journey and which contained the 5 1/4" cardboard floppy disk that was supposed to give me my morning hour back. Another week later I got a properly packaged and undamaged floppy in the mail. I handed in my notice a week later

when my boss decided to go with my original update plan so we (meaning I) wouldn't have to go through that again. Being the entire IT dept with no budget just wasn't me.

1
0
FAIL

NOVELL HELL.

At a soon to be failed insurance company the CFO learned that NOVELL 3.1.1 OS server did not have any backdoor if you supplied the wrong admin password then the OS would be on permalock and you will not be able to get any databack.

I had ZERO traning in NOVELL and had no desire to touch it. But the CFO wanted me to log into.

( I refused to touch it.) Found out the CFO wanted the server to have total dataloss so he could blame someone and not himself. And so was the end of PBHC health care.

Yes this was the same one that made headlines that patients getting medical support from teen girls who have no medical lic or background at all reading from Q&A books.

0
0
Facepalm

ISP in USA virginia.

The ISP who served Virginia,DC,Maryland ran their servers not in server or pc cases but just sat the motherboards on plywoood that had those industry fans blowing on them.

The server room had no locks.

Ticked off employee walked in and ripped out the server memory and people called tech support wondering why they can't connect to server via dialup.

0
0
Pint

another story from failed insurance company.

Same failed insurance company IT manager(had no IT training at all.)

Wanted me to build 5 servers from scratch and already had some pc cases laying around to use.

Walked back to manager to inform him the pc cases were too small to fit the motherboard in.

He did not care.... He replied ," MAKE IT FIT!"

Ok so I made it fit and had them all built and they all bluescreened and crashed or did not boot at all.

Because the case was so tight it was making the motherboards fault and sometimes bend and or touch the sides of the case and just die.

A multi certified DBA/network guru was sent to look at my handiwork and I told him what was happening and he was like really? So after he had a go at it they still failed.

IT manager wasted over $12,000 on things that did not work.

0
0
Anonymous Coward

And not even AC

Enjoy your lawsuit bro.

0
0

Page:

This topic is closed for new posts.

Forums

Biting the hand that feeds IT © 1998–2018