So, that vast IT disaster you may have caused? Come in, sit down

If you're good at lying you're FANTASTIC at everything else.

Normally the actual cause is not the person in the room.

Please Can We Have (PCWH) a test & QA system? - No Too Expensive (NTE),JFDI

PCWH Version control - Why waste time with that JFDI

PCWH have some training - NTE, JFDI

PCWH some budget for resilience - NTE, JFDI

PCWH a process to manage changes - NTE, no way you want us to specify what we want before you implement it? what do we pay you for?, JFDI

etc.

Oh yes we allowed them to interfere with your systems without telling you, after all they are corporate - why is it broken what did YOU do?

Oh we changed our minds, can you guess what we want and do it for tomorrow?

Oh that's not what we wanted, I know you wrote loads of specifications but we didn't read them we want something different - that one really gets me.

Now the times when I have convinced them that we need to do it properly all goes well. today for instance our SAN had a module replaced but the users never knew. four years of five 9's and counting!

City headhunter ..?

Yo Dominic, my man! Any generously-bonussed shit-will-never-be-my-fault-if-it-goes-wrong too-well-paid-to-fail jobs going?

Or does "headhunter" on this occasion mean you have the contract to hunt down the RBS wrongdoers and display their heads on spikes on the roof of Canary Wharf, pour encourager les autres?

I can dream ...

Cockups happen

I have made plenty of cockups in my career, luckily none of them happened in an operating environement. Doing stupid stuff like DELETE * FROM CUSTOMER_DB ; WHERE CUSTOMER_ID = 5 ; (see the misplaced semi-quote, you just deleted the whole database) are much less likely to happen when you have two people, one to check the work and one to signoff. Errors get picked up BEFORE they impact the live environement, that is why you have senior developers and change management. They are there to gate keep the idiots under them.

Problems can occur for any number of reasons, but not checking what people do (ie slashing resources ) and given them root access to your o/s or god privilages on your database seems like the perfect way to end up in sticky position. Often the root cause isn't disaffection, malicious behaviour, its just fat fingers and the right (or wrong) user privilages.

I have only ever once come across one deliberate attempt at sabotage when a developer tried to add a timebomb into some code. No idea why he did it (I think his contract was coming to an end and no renewal possibly) because it was picked up when someone QA'd his code. I have also heard of code being altered to adding malicious code to extract card numbers, again picked up in QA. If you dont have checks and controls you will eventually come unstuck.

If you get to the stage where you have to call someone to interview you have already failed as a manager and deserve to be put out to pasture to find your next "opportunity".

Techys can't tell you the whole story...

In a previous role I used to have to conduct "Post Mortem's" after significant outages and major incidents. Normally I'd spend a hour or so asking everybody what they did when in order to build up a basic timeline, then I'd line up the teas and go trawling through logs to find out what really went on.

The two would never match up.

Even if an engineer's actions were good and correct, they're recollection would be imperfect and there's always a tenancy to want to put yourself in the best light, and when filling in the gaps of memory the tale becomes skewed.

Most problems were caused by honest mistakes made with the best intentions, it was rare for problems to be caused by recklessness or full-on malice. Disgruntled employees who could do such things, can rarely be bothered to put the effort in to either try do things wrong or right.

Honesty Best Policy

I have lost count of the amount of times I have written emails along the lines of...

X was my fault, this is what the mistake was, this is what was affected by the problem and this is what I have done to correct the problem. ongoing issues from this may be Y

I have never lost a client due to an error and believe me in 18 years in IT there have been some huge ones (errors and clients)

Union reps and other points

@Oninoshiko, if you ask for a recording then that is fine, I am against covert or enforced recording.

I have no problem with a union rep, indeed although I don't like UK unions I regard access to one as a basic human right. But that's irrelevant, if you ask for a union rep and you don't get one then the employer's legal position is undermined, and so you should get one.

Whether it is in your interests in a tricky question that depends on what sort of rep and the situaiton.

The moment you ask for a rep or a lawyer then in some managers eyes you have admitted that you did bad. That's not fair, sadly almost none of my work is in the land of "fair" though I hear the weather is better there.

If you're going for a defence of bullying or discrimination then it might help, but that is a serious escalation.

Getting a lawyer would be really very hard for the average ITPro. Firstly they ain't cheap and unless you regularly get involved in screwups you probably won't know a good one, or even be able to accurately judge their competence. Recently I helped someone get a lawyer for a big issue in their life that I knew to be good for the specific task, quote came in at £ 1-3K

As for the anonymous coward who compared me to a faded film actor, that hurt, however when I dined in the Savoy last week I had no problem getting a table and if you will take my recommendation the pork was excellent. I guess that promotes me from faded 80s actor to faded 90s ?

What kind of job is that?

Forensic IT Interviews can only give you facts and evidence. In a modern business environment that's something very unpopular. Lack of facts gives you freedom to do whatever you think is right, unconstrained by empiricism. If a manager would have to act by facts and logic he could be replaced by a computer, or the next best person. He'd simply have to choose the best solution. The more facts a manager has, the less important he is.

The only reason I can imagine a company paying for that is to create the appearance of something being done. In that case the result is utterly unimportant.

I prefer...

..being the guy that comes in after the cock up. My politics are better when I can describe, solve, and lessen the wrath on the poor soul who made the screw up.

You techies

have it easy

Oh dear I've just deleted the entire customer database, the boss is going apeshit, the headhunter has been called in to take heads... oops conduct in depth interviews to get to the bottom of the missing semi colon and fire the person responsible.

In my line of metal bashing, oops we've missed a crack in a hydraulic coupling casting and there's 1000PSI oil going everywhere and the aircraft's rudder is stuck pointing 5 degrees to the left, not a good thing at 24 000 feet, or in setting up a new robot cell, we miss out verifying the door interlock switch in the PLC, some poor fool opens it while the cell is running and gets grabbed by the robot and stuffed into a machine tool.

Then the brown smelly stuff really hits the fan.

Along with the guilt of "did I f*** up and kill someone"

Another great article

I wonder if I'd enjoy these pieces as much if I wasn't retired.

Mind you, I wish I'd had Dominic around when my "retirement" was taking place...

Lies can be unfounded assumptions

I find the greatest hinderance to getting the truth is often an inability to involve the correct indiviuals in a proper evaluation. In some ways the article reassures in that it states that "binary" answers are not provided. However, I have concerns that an individual is engaged in this activity. When things get to the point described in this article then checks and balances are absolutely required. On a number of occasions in the past I have been asked to explains actions that I have not taken. Why ? Well because an individual with an inflated opinion of his own knowledge and interlect has made a critical assumption early in the analysis and then proceeded to complete the story on that basis. In all cases I have told the involved management that I would not accept any mopre instances when that individual was involved in any review without my participation.

Am I missing something?

My impression from the article was this is about determining whether it is likely to be accident or malice?

I realise the two are hard to separate but, as others have alluded to, things like having somebody else check your work mean you're less likely to the the guy who accidentally removed a disk from the RAID5, went to fix it and hot-swapped the wrong drive, rebuilt it, then tried to restore from a tape you accidentally kept appending to for ten years and have been storing next to a speaker.*

If Dominic Connor's job is merely to "find who is to blame" then it is, indeed, a very sad reflection of corporate culture and a chilling reminder that IT staff are not the ones who get golden parachutes. However, if he's called in to give some indication as to whether an individual responsible for a chain of cock-ups is unlucky, an incompetent liar or bent on destruction I'd say there's less to fear and more articles like this are needed.

*yes, I know, data centres aren't like that.

Yet more replies from Dominic

@Adamwill: Unions *were* a good thing, where we part company is that they are all that useful today given their tendency to follow agendas that workers either don't support or dislike.

As for the point about outsiders coming in to mess with staff being stopped by unions, the reverse is observed to be true. If you look at the public sector, much of which in unionised consultants crawl all over that assessing staff and "improving" their work practices.

Also they do play politics, one guy I used to work for was terminated because a union rep became his manager and they'd fallen out because of a strike.

The word "blame" is tossed around in this discussion a lot and references made to the more formal processes found in engineering. This part of my work exists because most IT groups have no such process, they get me instead. It follows that sometimes I have to work out whether the thing that happened was bad at all, one report I wrote basically said "I can see why this scared you shitless but actually nothing much really happened and it is gone now".

Backup tapes do fail, I recall reading various stats about how often backups can't be restored, and numbers vary from 10 to 50%. The vast majority of s/w bugs are stupid, not just mine either.

One issue in some code written by IBM was where someone had taken an error code, coerced it to a memory address and copied data into it. I will admit here and in public that I've called the wrong function by mistake. That's the nature of IT.

A big difference between safety-critical systems and corporate IT is navigation around black boxes.

Our daily work consists of dealing with large lumps that every so often fail to do what we expect and give diagnostics of the form "Operation failed". Which is less than useful and isn't predictable and where the vendor will flatly refuse to tell you why.

I have read the RBS is suing CA, someone like me is going to make some money out of just this sort of situation.

@joeman, if alls fair in love in war

then your manager (who has access to his manager and whom you don't have access to) is quite within his rights to say that he does not have confidence in your ability the next time a prime assignment comes up. He has not lied, he has just inferred there is a short-coming in your skillset (personal or professional). 2 or 3 of these inferrnces and you get a bad reputation without even knowing its happening.

Access is power, unless yours is better than his, you will nearly always come off second best.