Feeds

back to article So, that vast IT disaster you may have caused? Come in, sit down

The RBS computer fiasco gives me an excuse to write about a sideline I have in interrogating IT professionals who are suspected of doing bad things. Sometimes it is quite hard to objectively tell the difference between incompetence and malice. In fact it is rare that either are the root cause of the worst screw-ups. The most …

COMMENTS

This topic is closed for new posts.

Page:

Silver badge

"The problems are being caused by him or her trying too hard"

With the boss standing right behind yelling "WHY ISN'T IT FIXED YET? HOW LONG IS IT GOING TO TAKE? THE CUSTOMERS CAN'T ACCESS THE SYSTEM! HURRY UP. AND GET IT FIXED! WHY ISN'T IT FIXED YET?

Odd that sometimes they can' concentrate properly.

44
1
Silver badge

Oops that should be: can't concentrate

1
0
Anonymous Coward

It's the name of the game

Cool under pressure is the name of the game. When I used to work hard, I used to play hard. Repairing financial systems with branch managers wanting their systems back up and running. Some were better than others.

A few years ago, I exchanged the pressure for less money. Net result; probably less liver damage in the long haul.

Money isn't cheap.

8
0
Thumb Up

Nothing less likely to produce good output than a flapper standing next to you.

4
0
Silver badge

Oops that should be: can't concentrate

See what happens when you are under pressure*

*I have now read the above three times to check it says what I think it says.

2
0
Anonymous Coward

@Christoph

Standard answers to this sort of interefering, no matter who it is from

>WHY ISN'T IT FIXED YET? - Some idiot keeps interrupting me - once you've said that to your bosses boss a few times and you're still around you know you're valued by someone even higher up, however you still won't get a pay rise.

>HOW LONG IS IT GOING TO TAKE? - This is my favourite question and my stock answer is, a lot longer if people keep asking how long is it going to take. It won't stop them aking

>THE CUSTOMERS CAN'T ACCESS THE SYSTEM - And? As if I didn't know and even more presumptious, as if I cared.

>HURRY UP. AND GET IT FIXED! - I take a breather, the quickest surest way to get me to slow down is to tell me to get a move on and the quickest way to get me to stop is to look over my shoulder. No spectators, if someone is going to look over my shoulder they can do the work while I do something more entertaining.

And yes, I do use these responses, I also unplug the phone, what's the worst that can happen? They fire me, as if I'd be upset. Panicking and dropping your trousers never solved anything.

13
0
Devil

Re: @Christoph

One day I was fixing a problem (don't remember what, and doesn't matter). And this guy was "Go, go, go! Fix it already! I can't wait much longer! Hurry, hurry, hurry!"

Then I looked to him, and said: "Slowly is faster."

I could see him changing colors... :D And 5 minutes later the problem was gone.

2
0
Anonymous Coward

My line with the boss...

Which would you rather I do: get this fixed, or talk to you about it?

4
0
Boffin

A couple of times I've been roped into fixing computers for friends, who then proceed to hover nearby and do an impression of Stan Laurel just after he's done something untoward to Oliver Hardy while you're trying to think. Obviously you can't just tell them to go away, but if you ask them to put the kettle on it gives you a few minutes to get started. When they see you with your head down getting on with things they tend to leave you alone.

0
0
Anonymous Coward

Re: @Christoph slowly is faster

Ah Master your wisdom is profound.

0
0
Bronze badge
Coat

Re: @Christoph

>HOW LONG IS IT GOING TO TAKE? - One answer that has worked for me is, "Boss, imagine you've lost your keys. How long will it take you to find them?" Of course, I made damn sure I knew what the problem was before saying that, and all was well a hour or two later...

1
0
Gold badge
Happy

"A couple of times I've been roped into fixing computers for friends, who then proceed to hover nearby and do an impression of Stan Laurel just after he's done something untoward to Oliver Hardy while you're trying to think. Obviously you can't just tell them to go away, but if you ask them to put the kettle on it gives you a few minutes to get started. When they see you with your head down getting on with things they tend to leave you alone."

If only this would work on the IT Director!

Although I suppose you might get some respite, while he's recovering from the shock...

I suppose there's always the BOFH option? Cattle-prod, and lock them in the tape safe.

0
0
Anonymous Coward

Re: How long is this going to take?

Oh, my boss loved this one so much he used to rerun it over and over...

I don't know, I've never done it before

BUT... If the thing in question is unscrewing one's own screwup, then a slightly more accurate answer is demandable.

0
0
Boffin

Oops that should be: can't concentrate

Or...can't concatenate?

0
0
Silver badge
Happy

Re: My line with the boss...

I never had to work in the financial sector, but I did work at the University hospital. Whenever one particular senior staff member (non-computer science) tried to meddle in my work and ask what was going on (with an implication that things weren't going fast enough), I tended to explain in detail what I was doing, with as much mathematical jargon as I could muster (which is a lot when you are doing image analysis), in a friendly, fellow-scientist-to-fellow-scientist tone (i.e., I do not have to explain every concept in detail, do I).

The first mention of "fast Fourier transform" or "optimal convolution kernel" was generally enough to make them run for cover.

0
0

Tape?

Lovely piece.

I wonder if Connor records these interviews. So many times I've had a sensitive meeting and then thought "did he really say that?"

2
0
Anonymous Coward

Re: Tape?

Any meeting like that I'd be recording it on my phone myself. Same thing I'd do with any non-trivial interaction with the police, or TV licensing inspectors. Might not reveal I had the recording until after they'd purjured themselves giving false evidence, mind...

7
0
Bronze badge
Headmaster

Re: Tape?

eh, if you didn't have the permission to record you have no usable evidence.

You never noticed those "this phonecall may be recorded" jingles before, I take it?

1
1

Re: Tape?

You'd be very surprised at what can be entered as evidence and, more germanely, what might well prompt an out-of-court settlement.

6
0
Bronze badge

Re: Tape?

In the UK evidence obtained illegally is still evidence. Unlike in the US, where I understand that evidence obtained illegally is not evidence.

2
0
Rob
Bronze badge

Re: Tape?

Even if it's not usable as evidence, spice it up some more and mention you have some journalist friends or that you work in broadcast media, you'd be amazed at how quickly attitudes soften enough for your purposes.

1
0
Headmaster

Re: Tape?

In some countries it's perfectly legal to record a conversation you're part of, even without other parties knowing about it. In Finland, for example, where it was actually confirmed by the (Finnish) Supreme Court some years back. And yes, obviously such a recording would be usable as evidence.

0
0
Anonymous Coward

My manager won't allow me in certain meetings anymore, especially those involving those above him because I refuse to lie.

My Nan, the most passive and gentle soul on the planet went into one, once and only once because I lied to her. Watching her losing it frightened the living shit out of me, I find it incredibly hard to lie outright.

You lie and people find out later on they're more pissed off than if you just said in the first place, "Sorry I fucked it up and I'm fixing it, OK!". Bending the truth within the context of what I have to work with, yes I can do that but lying I can't do, I just can't pull off the required poker face. I will forever be a techie on the factory floor, no giddy heights of management for me, that suits me just fine.

24
0
Bronze badge
Pirate

Warning ......

I have had at least two managers who have suggested that any communication should go via their good selves. The usual phrase is 'The Team should provide a consistent message.....'. In my experience this is the time to get a new job. Not quite sure if this the same sort of situation you are describing above Mr AC.

2
1
Anonymous Coward

Re: Warning ......

Or "my job is to protect you from the directors". Ohyeah, why would that be then considering I am competent.... perhaps other people needed to lay off the Bolivian Marching Powder? ;)

1
0
FAIL

Re: Warning ......

I did an internship many moons ago where I was tasked with reviewing use and integration of a minor piece of in-house, front-end software at an international bank..

After asking me to re-write and expand the report multiple times to include ALL the problems AND solutions, the bumbling idiot shifted it straight up the command chain to head of IT without proofing it.

I get called up, a personal letter of recommendation from the head, and offered a job when I graduate. I have a meeting with the bumbling idiot where I am told that "In these reports, one shouldn't be too blunt or it could cause problems."

He was fired within the quarter.

Some people need to protect from themselves.

0
0
Anonymous Coward

Re: Warning ......

Heh, had that a few times. Funniest was when sent out on deployment for another department. Had my programming boss, the professional services boss in the UK, the foreign department boss, boss of the company, and various leeches throughout the company all demanding that I send THEM the daily diary and that THEY would be the one to use it to bash the others with. I spoke to the mail admin and asked him to set up a dailydiary@comp... address, and that I'll send the reports to that and that only. Then the managers can all argue amongst themselves who should get access to that list.

Wasn't long before I had the emails telling me who wasn't considered important in the company as they were asking for the diary before I sent it to the agreed address, just to check it was factually correct.

Bunch of donkeys. 2 mangers+ for every actual worker, with many people reporting to more than one manager. Wasn't too obvious how badly it all ended up.

0
0
Anonymous Coward

Re: Warning ...... @Asiren

Reminds me of a time when I was asked to do some work for a 3rd party company. It was checked and re-checked for bugs and was then sent to management to be sent on. Not long after a red-faced manager comes steaming in screaming about how he'd sent it to the customer and it didn't work and how it was an embarrassment.

There was a config issue relating to the clients computer, I'd fixed it in minutes.

It wasn't until I read the email trail that I found out why he took it so personally.

Oh yes, he said he'd written it personally. When the (techie) client asked him to fix it, he got monumentally rumbled.

0
0

Re: Warning ......

Not necessarily. I've done a fair bit of customer-facing work in the past, and one thing I've learnt is that significant communications (i.e. relating to deliverable content, timescale or cost) should be written, read again and fixed, and then read by someone else and fixed. A three-paragraph email may take an hour to write and send. This might seem ridiculously wasteful, but believe me, it's nothing compared to the cost of dealing with the fallout when you find that your badly-written email has led to a high-level flamewar.

As lots of people have mentioned, engineers aren't always switched on to the implications of what they say. Compare and contrast emails saying "we knew about this bug but we didn't fix it" and "we knew about this bug, we discussed it with your liaison Mr Jones, and collectively we decided it was a low-risk issue compared to feature X (see meeting minutes attached)". Eventually the first email will probably get all the relevant information out and the flames will die down, but you really don't want the grief involved.

Nor are engineers often very savvy about committing their company to things. An engineer's typical response is "sure we can do that", because technically they *can*. But it'll take an extra month of work, the customer's deadline is in six weeks time, there's three weeks of acceptance tests to run, and you don't have budget for that extra work anyway. Do you fancy doing unpaid work for your company?

2
0
Anonymous Coward

There is a trust option,...

should you tell the truth if you think it can be skewed by those who receive the report? That 5% malicious may be enough for them to say "its possible it was intentional" and bin you and hide the report (seen it done to others, been put on report myself using a similar tactic).

Dominic, the thrust of your recent articles seems to show how technicians have a difficult time with politics. Having also read "the way of the rat" by Joep Schrivers, I think there is a market for a "how it works" book about politics for techies. Fancy bashing one out?

7
0
Silver badge

Re: There is a trust option,...

Workspace Politics for Dummies, anyone?

3
0
Silver badge

Re: There is a trust option,...

Union reps have their use, especially when you bring them to these meetings. Having a union to back you up on dismissal does help. Especially when your union rep can shorthand.

5
0
Silver badge
Stop

Re: There is a trust option,...

The thought of having BCS representing me in any kind of official capacity terrifies me far more than the prospect of losing my job.

IT does run the world, it is a shame we cannot fully unionize. Bob Crow and the RMT think that tube drivers control London, imagine if all the techies went on strike.

5
0

Re: There is a trust option,...

bcsthecharteredinstituteforit is not a union. Prospect, on the other hand, is. Not sure why you think all IT people couldn't join it.

1
0
Anonymous Coward

Re: There is a trust option,...

There's no cure for mild autism.

That's harsh, but the vast majority of good techies genuinely have seriously under-developed social skills. They often also have this blithe attitude towards politics and are completely happy to call lesser mortals (who y'know.. PAY THEM...) idiots to their faces; which is never helpful.

That's why sensible managers don't let the techs go to meetings without an escort... or at all, if can be helped. You'll invariably find that the one extrovert tech with any skill in diplomacy in the department eventually gets saddled with every Change/client/crash meeting, which is the best solution for everyone. The techs who hate everyone can stay tucked away doing what they do best, and nobody gets hurt.

I'm not sure a lot can be done to help, in many cases. Learning to keep quiet and smile is probably the best advice that can be given to a great number of people.

11
2
Pint

Re: There is a trust option,...

" Learning to keep quiet and smile is probably the best advice that can be given to a great number of people."

Funny you say that. I almost got an employee of the month for saying about two sentences in a client facing kick off meeting. However, one of my co-workers almost got terminated for dribbling on for thirty minutes in the same meeting.

0
0

Re: There is a trust option,...

"That's why sensible managers don't let the techs go to meetings without an escort... or at all, if can be helped."

This would be fine with most techies. The problems start when people decide that not having the borderline/ narcissistic personality disorder needed to play office politics well, is some kind of objective personality deficit that should be rectified by exposure

0
0

Re: There is a trust option,...

If "the truth" puts the root of this at deliberate malpractise by you, I don't have any sympathy. But if "the truth" means saying who told you to do what, and the shortcuts you were required to take on testing, and showing the list of defect reports you raised, you're a lot better off. If this also includes emails, then great.

Amongst techies, rats have a very short half-life-to-detection, but most techies (especially newbies) aren't so good at dealing with them. The only good defence against a rat is to block up ways in. If his moan of "this was 2 weeks late" CC'ing his manager is replied to with a "sorry, but you added on extra work for which I estimated 2.5 weeks (emails attached)", you're sound. You don't need to be a rat yourself, you just need to stop rats from getting to you. Call it anti-politics, if you like.

1
0
Silver badge
Boffin

Auditors

"“Interrogation” is of course exactly the wrong word and if I’m dealing with a firm that has handled this sort of situation before, one that is utterly forbidden."

Indeed, it's "Root Cause Analysis" but in this particular instance I suspect that PWC or similar will be in to do audit of how the change controls allowed this sort of c0ck up. There are FSA requirements around this sort of control, but more chilling for RBS is the fact that they're also an NYSE listed company and are subject to the rigours of Sarbanes-Oxley compliance, which makes for an even duller and intense process.

1
0

Preparation is the key

Looks like good advice. I would also suggest you make sure you are really prepared before heading into any meeting like this. Make sure you have copies of key e-mails, have througly reviewed any files before the meeting and have prepared answers to any of the questions which may be asked. Don't be fooled by the convivial atmosphere, anything you say here could form part of a later disciplinary.

As the tone of the article suggests, the aim of such an investigation is to assign blame. Usually there will be enough to go around and you just need to make sure it isn't disproportionately dumped on you.

3
0
Silver badge

Re: Preparation is the key

offsite copies of emails too. Even if you arent allowed to copy emails then a good cached copy of your work account is good enough.

0
0

wow, who the hell . . . . .

This article appears to me as totally generic. What we (the once who know what they are doing) do, has nothing to do with luck!

The good boss hires only good people. Here we are hiring family and friends, wannabees or cheap slaves. No respect for the true value of the human resource.

RBS's cockup is the result of the lack of basic knowledge. In the real IT world things like this do not happen.

Neon

0
22
Anonymous Coward

Re: wow, who the hell . . . . .

Ah, an 'I never make mistakes' man.

I'm glad you've got this far in your life without having that one day where you did something that 99.99% of the time you would never do but slipped up once because you'd had a fight with the wife/gf and couldn't concentrate, were ill but chose to work anyway, etc. When that day comes, you want to hope that your one mistake is on something trivial because people forget about the 99.99% of perfection the one time it's not.

Not suggesting that the RBS thing wasn't the result of using people without the right skills but you are generalising a lot. It's always someone else who knew much less than they should have. Until it's you.

14
0
FAIL

Re: wow, who the hell . . . . .

in the real world stuff like this does happen... you still stuck in the matrix?

1
0
Anonymous Coward

Re: wow, who the hell . . . . .

There are two types of people, Neon. There are all those chumps that make mistakes, and there's me, who also makes mistakes.

(Though on a good day I spot them before anyone else, and fix them without telling anyone.)

To all those people who make mistakes. How crap are you? I'd give up work were I that bad.

</irony>

2
0
Anonymous Coward

Re: wow, who the hell . . . . .

"The good boss hires only good people. Here we are hiring family and friends, wannabees or cheap slaves. No respect for the true value of the human resource.

RBS's cockup is the result of the lack of basic knowledge. In the real IT world things like this do not happen."

RBS's cock-up is not the scope of the article. Go comment on an RBS article of you want to drag that into it.

And in real IT world, sh!t like this *does* happen. Sh!t happens because people are tired, rushed, miss a line on the screen, suffer cognitive bias or blindness, and a hundred other reasons.

A tech who has never taken down a production system and packs the attitude that he never will because he's 'too good' is a dangerous liability, to my eyes.

3
0
Pirate

Re: wow, who the hell . . . . .

Yep, one sign of "maturity", both as a person and as a computer nerd/professional, is the ability to recognize those days when you shouldn't type "rm -rf *" for any reason whatsoever. Days like that happen, and when they do, you're sure to fuck up in ways that will make you miserable. Better to realize it's gonna be a bad day before it becomes a Bad Day (tm), and avoid doing heavy-duty stuff. Make it LOOK like you're doing Stuff That Matters, but in actuality you're just testing all the various switches to "ls" and writing scripts to send email to yourself.

5
0
Anonymous Coward

Re: wow, who the hell . . . . .

I like to think that I'm one of those who is above screwing things up - at least in production. I take the time to do it in staging... case in point - rebuilding vmdks on a system the other day. Had to be done as a rebuild in parallel and swap out of the drive via e2label etc. Got all done. DELETED THE WRONG BLOODY VMDK FILE... At least it was in staging but still took out a day of work and another day trying to coax the VM back to life

Ask the people here and they will tell you that I never fsck things up, know wall the obscure commands on *BSD,Linux,Windows, MacOS etc.. But truth be told even then you can still screw things up. Its just a matter of where you do it and who it impacts that keeps the depth of the crap down sometimes.

(Luckily not had a production version of this yet - but I know one day I wont be so lucky).

1
1
Coat

Re: wow, who the hell . . . . .

Or you could be the kind of person who thinks too hard about what might go wrong and how you could cock it up, even though 99.9% of the time you won't, and then you don't do anything, even though you can.

That's when you learn the trading floor ain't the right place and you move on, happy in the knowledge that those people who are earning more than you for being no better, do actually deserve it.

I wouldn't go back, but I'm very glad I was there for a while.

I'll get my coat, because I did.

0
1
Facepalm

Re: wow, who the hell . . . . .

Don't be too hard on the poor little mite, we all thought we knew everything once.

3
0

Page:

This topic is closed for new posts.