'Numpty new boy' lets the boss take fall for mailbox obliteration Welcome once more to Who, Me?, El Reg's weekly confessional column for readers with technical mishaps weighing on their minds. This week, we meet "Sam", who made a rather big error when following what was, in his office, standard procedure. At the time, back in 1999, Sam was less than a year into his IT career and worked at a …
I've haven't let people take the blame for my mistakes before, but I did once see my boss struggling desperately to save another of our team. We needed a switch replacing that was only used by IT staff for bulk-imaging laptops. The old switch was never properly registered on our systems, so our friendly network engineer offered to replace it without raising a change request (there was no original switch to change officially remember).
10 minutes later, I lose network access. So do all my colleagues, and I see the Cisco Wireless AP with a red blinking light on it. I popped into the room next door, just to casually inquire if our friendly network engineer knew that the entire building's network seemed to have died. Cue a barrage of "Fuck-Fuck-Fuck" and an emergency yanking out of the patch cable from the new switch our engineer was configuring.
We were honest when explaining to our boss why he'd just experienced a 5 minute outage (he wasn't too impressed, but accepted the profuse apologies). Knowing that the executives upstairs may not be quite as forgiving, he had to then burble a number of excuses upstairs to try and put the matter to bed.
"What do you mean all ownership on all homefolders is gone!"
Scene: picture a bitterly cold and miserable European winter. Boss decides to take his new girlfriend to a sunny Middle Eastern country for a remote office deployment rather than send someone more technical... The remote office requirement is fairly basic to allow access to central finance systems as new projects reach maturity and provide connectivity to HQ as the basis for more services in future. Things were a little rocky when he was unable to get the remote end of the site-to-site VPN up due to "ISP issues" which also significantly limited the rest of the teams ability to assist. Towards the end of the fifth week of a two week trip where 4-5 hour calls to the boss were standard:
What do you mean you made all 45 users Enterprise Admins and they still can't access the file shares you created on a new server?! Why?
... If it is well known that you know which closets have skeletons, and that you know where all the bodies are buried, and you can keep your mouth shut about it ... THAT is when they start paying you the big bucks.
Or, as Great Grandpa said: "'Ear all, see all, say nowt". (He came across the country to California by covered wagon during the gold rush ... he started learning English during that trip from his first wife, a Plains Indian, who had learned English from her first husband, a Yorkshireman.)
I know this is an amusing look at mistakes people have made, not an advice column. That said I want to mention that in my time in IT I have seen four people sacked because they made mistakes and lied about it, or tried to cover their tracks. I've never seen anyone sacked who made a mistake and immediately came clean. Maybe I've been lucky not to work anywhere with a culture of blame but I don't think so. So my advice is immediately come clean, unless you are the only person capable of understanding your mistake. In that case go nuts, blame fluctuations in the Earth's magnetic field.
I once worked for a company that had a total blame culture (one of the reasons why I suspect it failed later).Not realising this when I started, I made an error when configuring the settings on a new LMS. When it was noticed, the cry went up "Whose fault is this!!!" I turned my chair round and said "Whoops, I screwed up." You could have heard a pin drop... They were so shocked that nothing else was said. :-) Mind you, in the same company I did threaten to defenestrate (from the 9th floor) a colleague who tried to pass the blame for a really critical error on to me.
A true BOFH would've chucked the offending miscreant, claimed the expenses for the damaged car, and the expenses for the mileage of driving to and from work, despite regularly commuting via public transport.
With ironclad witnesses as needed. Like his buddies Charlie Ash, William Onga and Samantha Pondu-licks.
"blame fluctuations in the Earth's magnetic field."
I once worked hell desk and had a corporate client who often called complaining about network connectivity, Our team soon deduced it was caused by thunderstorms at the client office, but it's hard to tell people that on the phone, even when true.
You can only come clean if you know you screwed up. Sometimes we screw up and don't know it particularly when writing scripts. In my case, it was decades ago and we didn't have a test server, only live data. I hiccupped writing a script and it didn't work. Reworked it and it worked fine. A bit later, the database started having issues. We shut it down and rebooted the server and everything was fine. After head scratching we traced it back to my script (the version that "didn't work") and a mistyped command. Luckily the boss was very understanding and we had our test server pretty quick after that.
Luckily the boss was very understanding and we had our test server pretty quick after that.
I've "accidentally" done a few things like that in the past, when the reasons for a test (or backup) machine weren't fully clear to the boss.
Completely accidental and only due to the stress of NOT having a safe/redundant way to test things without error, of course.
(Funny how after I got the test gear I never managed to make such mistakes again, must be due to the relief from knowing I wasn't putting the company's data at risk... :) )
When I was the chief architect of the second Dartmouth Time Sharing system I had an agreement with the operators to tell me exactly what they did when they screwed up as this information was invaluable to recovering from their mistake. Sometimes it was a system problem inadequately presented to the operator and other times it was a genuine operator error. I made sure that none of the operators got in trouble and recovery was made much simpler.
I like the term, even if it depicts the most toxic work environment.
I now work for them, mind you, but being the cynical git I am, I only cash in and don't mind the toxicity.
Fuck them all, I'll retire soon.
But frankly, I'm just wondering where is all the ethos of PMP, ITIL and the rest, in the current IT landscape.
KYB - Know Your Boss
As in, if you cough up, will they throw you under the bus, or will they protect your arse while you fix stuff?
As an occasional boss (although hopefully not a PHB variety) I work on the basis that if you tell me you messed up my job is to run interference for you, if someone else tells me I'll wring your bloody neck. This does occasionally include the following sort of conversation:
Them - we broke something, we're going to fix it
Me - what are you going to do?
Them - you don't want to know
Me - okay
I believe in politics its called plausible deniability.
Yup, thirded on this approach to management.
In my last management role, it tended to be my guys made a mistake and owned up, i covered for them and my boss happily threw me under the buss with the directors.
He was quite miffed when myself and the entire service desk all quit on the same day.....
At a company I worked for in the early 00s, everyone's password was 'password'. Except for the partners, who could set their own passwords, which we then kept a copy of in an Excel spreadsheet.
Soon after joining I was asked how to improve this - at which point I showed how we could and should configure password complexity and maximum age, etc. via group policy, and I'd be happy to implement it, which would only take five minutes - but we should communicate it first, so maybe roll it out over a couple of weeks to give people time to adjust.
When I left that company five years later, none of my recommendations had been implemented. Everyone still had 'password', and the spreadsheet of partners passwords was still in use. My manager insisted on keeping the status quo because it made support easy when he could log on to anyone's account to fix their issue.
I made sure I covered myself with an electronic trail of all my recommendations and my manager's refusal to implement them. When the company created a compliance team a few years into my tenure, I also flagged it with them. The biggest surprise to me was that this was all ok with the industry auditors (insurance industry), because we were 'working on a fix'. The fact the fix was ticking a few boxes in a GPO and we hadn't done it after years of working on it didn't seem to bother them (or more likely, my manager was 'creative' in his explanation of the issue).
Anon because of shame.
I've found it better over the decades to hold my hand up straight away and admit when I've screwed up.
The two big advantages that I can point to:
1. You can get on with fixing the problem straight away, rather than waiting until somebody else notices that there is a problem.
2. When something goes pear shaped that isn't your fault it's likely that the powers that be will believe you when you say it wasn't your fault.
The smaller advantage is that you don't have to keep a catalogue of pork pies straight in your head, so you never get caught out by saying something stupid later.
Same strategy. Errors happen. Most of us are human.
If someone else fesses up (even under confidality) as senior admin I will do anything possible to solve the problem and to keep the impact minimal. There are several project managers amd admins that owe me their jobs. With them I can work together, because I can trust them to tell me what really happened.
If someone tries to shift blame instead of solving I will not cooperate and if someone actively blocks attempts to solve the problem to cover his/her tracks I will refuse to work with him/her together further on. Openly. That person cannot be trusted with admin rights and projects are doomed.
That's always been my policy. When I make a mistake, I own up to it loudly and immediately. If it's not discovered immediately and it's brought to my atrention, I take full responsibility for it loudly and on the spot. Because of this, when I say "Nope, I didn't do that" it's accepted without question.
Ha, Good post. I hate this 10 min edit limit....
Thankfully I'm non-techy so any typos I make have limited result. I am aware of several of the bigger mistakes made over the years, and the measures taken to cover them up, but that's because I'm friends with the right folks in IT.
“We had Windows NT 4.0 workstations for the most part, and the user accounts all used roaming profiles”
I've never seen ‘roaming profiles’ working even when they did work, minutes waiting for the profile to be copied down to the client and minutes waiting for the client to be copied back to the server usually because some process is keeping NTUSER.DAT locked.
“NT 4.0 just over 18 months before, and they were seeing stability issues, especially as it related to the Novell login and roaming profiles.”
Wonder why that was:
‘On 05-21-98, I called our Microsoft Premiere Support number to request help with the conflict between GW and Outlook 98 .. I got a call from Adam. He told me that MS views the way Outlook 98 was operating as a "Feature", not a bug’ ref
“Below is the text of 2 messages sent previously regarding header files and libraries for implementing a Windows 95 Password Provider. To date, we have had no response, but we need this information” ref
“My MAPI service providers that used to work in the M7 time frame (January beta) no longer seem to work.” ref
“It should be noted that these bugs, for the most part, are not problems with our software (the Win95 bugs are problems we addressed with Microsoft which they refused to fix).” ref
I've never seen ‘roaming profiles’ working even when they did work, minutes waiting for the profile to be copied down to the client and minutes waiting for the client to be copied back to the server usually because some process is keeping NTUSER.DAT locked.
THIS.
In my last job supporting schools we used them for the teachers and not the students.
We also redirected folders like the desktop to be on the network share to minimise the size of the profiles, and I rarely had any issues, and most of them were down to the teachers trying to store multiple ripped DVDs in strange folders that were part of the profile.
In the 1980's the large-ish company where I worked trotted out a new email system. Many members of management were unable to adapt and continued to dictate messages to their secretaries to be sent by email with a typed copy placed in a file drawer, followed by an interoffice FAX of the identical message, followed by the secretary hand carrying a printed copy of the email to the recipient's office. The behaviors improved over time, but is still amazes me the company was profitable.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
