Feeds

back to article National Grid computers locked-down in outage cock-up

National Grid suffered a major computer system outage last week, leaving many of its staff twiddling their fingers while services supplier Computer Sciences Corporation (CSC) scurried to provide a fix. The electricity and gas operator, which supplies energy to homes and businesses throughout England and Wales, confirmed the …

COMMENTS

This topic is closed for new posts.
Stop

This is why Windows has a bad rep....

twats like these.

How on earth can you 'delete AD'.....? Surely there could only be a couple of people who have access to the actual database file itself. Are you telling me that they deleted the database from the cluster?!?!? WTF were they thinking!?!?!?!

Alternativley some pissed off employee decided to delete all the objects in the directory... which makes you wonder if they have heard of delegation or do they just give all helpdesk staff domain admin rights....?!

And how the fuck can it take more than a couple of hours to restore the database...? And as for email, all you should need to do is reboot your Exchange cluster once AD is back up and voila.....

Jesus - and companies pay these people millions to look after mission critical IT. You couldn't pay me trust them with a fucking toaster.

0
0

And we're gonna trust them with a national ID card database?

Hmm....

0
0
Alert

This is what you get from Outsourcing, even to UK outsource companies

You get what you pay for. Cheap=shit.

How on earth could anyone but the *very* few Enterprise Admins, who should have the knowledge of God himself, be able to do such a thing.

And where was the backup and emergency plans and procedures. Probably lost in the mess of outsourced dept muppets, contacting outsourced support desks, who refer them to "my mentor", who ask them to log a support ticket before they'll lift a finger.

Its bloody terrifying.

0
0
Thumb Up

CSC

They really are that bad, having to work with them on a regular basis it's taken them nearly a year to approve copying some files onto a server. No matter how bad you fuck up you will never look as bad as them...

0
0
Stop

Where have I...

heard this before?

Oh, yeah a similar thing happened to us at Network Rail over the weekend. It wasn't all of AD that went missing, just some of the tabs...

0
0
Bronze badge

Right...

Um, yeah. I have to agree. Its not exactly that easy to completely kill AD, especially by accident.

Even IF you did have someone deliberately kill the system surely it doesn't take 24 hours to restore from your previous days backup tape...?

More importantly, are the same people responsible for the systems that are managing the power grid?

0
0
Silver badge

To Osama...

...don't bother, we're doing a great job on our own.

0
0
Silver badge

Not surprised

I was a contractor at NG doing some CAD work a few years ago at their HQ. Put quite simply, CSC didn't know their arses from their elbows.

I was only there a couple of months, but it seemed to me that the vast majority of the IT bods had just been on one of these "earn more by working in IT" training courses, and thought they knew everything. And the helpdesk bods were worse than those at Orange broadband (and that's saying something!!!)

In the end I told my boss I wasn't gonna put up with their ineptitude and forced them to pay me extra to use my own laptop for the job. Even though I had to transfer files on floppies, and the laptop was old and on it's last legs, it was quicker than waiting for the ***kwits to get anything done.

0
0
Anonymous Coward

Not a great surprise

Having worked for CSC as a PM,on implementations and at NGT, they are very quick to give out "cloned rights"

The concept of admin rights is always seen as "needed to get the job done quicker" rather than a reduced delegated rights model of least privilege.

The open door "complain about security and get pushed out of it" is an excellent idea for bean counters.

I had auto forward emails still being sent to a hotmail account nearly a year after leaving, makes you wonder doesn't it.

0
0
Anonymous Coward

Break it down - worst case.

Ok lets think about this (therory)

server dies (fire etc)

incident raised 30 mins

engineers travel to site 2 hours (heavy snow in solihul)

rebuild of server cluster 4-6hours

setup of temp FR inc certificates 30 mins

install restore agent and restore database 1-2 hours

send out a tannoy (lets pretend the AD was D/R across the infastructure at different sites) 15 mins

reboot every infastructure piece of kit 2 hours

so thats your day gone!

more likely:

time to raise incident 1 hour

call passed to wrong team 29 mins

call passed to wrong team 29mins

call passed to right team who smile and go for lunch 30 mins

team calls the account manager who rubs his hands together 5 mins

engineer goes "phaaaa its a big job" and drives to site 20 mins

engineer on site sits for 30 mins in reception

engineer gets to server room, boots his laptop from a pendrive as the AD master server, restore the AD database to the AD cluster, promotes cluster to master and shutsdown laptop calls account manager 1 hour

2-3 hours later the account manager get back from his golf lunch and tells the customer the good news - 1 day and sells them a uk wide resilient DR solution that they already had.

0
0
Bronze badge
Unhappy

How did this happen...

Um, so, they have AD, which natively supports, and is designed for redundancy.

Are you seriously telling me, that someone managed to hose all the AD servers, and all of the GCs? ?!?! ?!?!? What in the name of god are these clowns doing.

0
0

you pay millions to set up the network

and minimum wage for the admin staff and this is what you get.

hire some qualified admins...

0
0
Coat

24 hours...

Did it take them 24 hours to restore the backup or did it take them 23 hours of that to work out how to log back in and get their admin rights back so they could restore the backup?

Wonder if they had to physically take the backup take and the admin to the machine. Outsourcing, you've got to love it.

0
0
Paris Hilton

CSC Clowns.

Why didn't they restore from backup? Because they didn't have one. Nor is this the first time something like this has happened and the CSC cretins have had to admit their backups haven't been running.

Frankly I wouldn't let them near my home PC let alone anything else.

Paris, because she'd almost certainly do a better job.

0
0
IT Angle

Saw the same thing in local government once

Different company, but the scenario was as follows:

Onsite AD admin, bloke barely knew his way round a windows desktop, his career to that point had been Novell - He's on the blower to a 'trusted service partner' (Guy at 'large PC builder' doing support while playing solitiare)

"So whats the easiest way to reset permissions on the desktop accounts?"

"Uh-huh so after I delete them?"

"Backup? How do I restore from that"

Cradles phone for a second...

"Any idea where the backup for the AD box is?"

Cue silence....

Swiftly followed by the account manager going very pale as he realises the penalty fees for loss of service, which accumulate hourly.

"So how do we recreate accounts for all the machines in the schools across the city?" he asks

Me and my cohort look him in the eye and come up with

"Well, if you had the computer accounts in a spreadsheet or the like we could try and knock up a script, but you'll still need onsite to rejoin each machine to the domain from their side"

"But it's a bank holiday... and we don't even have that list"

I was asked to leave the site for not being a team player - I said no when they asked me to drive round 20 odd schools on a bank holiday weekend.

Anon as anyone who was onsite that day will recognise it immediately

0
0
Anonymous Coward

Expected from a privatised state utility

The National Grid company no doubt has a total monopoly over the electricity wires in this country (or at least the big-uns). I do not think that utilities should be run as for-profit enterprises, because ultimately the business will be run for the owners rather than society it is meant to service. Obviously when the state run these utilities they become lumbering great money pits, but when private enterprise runs the utilities the utility becomes a money spinner for a few, then require lumbering great state regulators aswell! I don't know how utilities should be run... maybe as some kind of non-profit organisation?

Basically, NGC would have got their IT services from the cheapest bidder: CSC. It is no wonder that NGC end up having a major outage because they have contracted in clowns [1]. They go for the cheapest bidder because the people making the financial decisions don't have a clue about technology, but do have a clue about their bonus if they hit their targets (stuff like implementing supported computers for cheap).

[1] The only difference between any of these IT consultancy companies is how well the sales staff can lie to the prospective client: In the end, the consultants will charge more after the contract has been signed for services that the client needed and was never sold, or services will be sold to them (as mentioned above).

0
0

The scary bit...

... is that if this is what it looks like when you're safe, imagine what it looks like when you're in trouble.

0
0

@You couldn't pay me trust them with a fucking toaster.

actually, you pay them (indirectly) to keep your toaster running!

or not running as the case may be...

0
0
Silver badge

Oh god...

"One source claimed that Active Directory (AD) clusters on the energy giant’s Windows 2003-based server were deleted late last Tuesday. "

So they're using Windows there. I hope they're not using it to actually *run* the power grid!

Then again, back in 1993 one hydro powerplant went offline here in Mexico, taking down power in *nine states*. Why did it go offline? Lightning struck one critical piece of equipment. Oops!!!

0
0
Flame

computers are crap anyway

Once the user has all their stuff on it then they are yours to bill as much as they will accept. It's only word documents and email, nothing important.... Oh wait wrong thread, it's the UK's Electricity supply, if that goes off, all computers crash!!!

0
0
Alien

Sadly

Sadly , we were warned about this ever increasing style of management incompetence back in 1969 !

0
0

Probably...

Baby firewall admin saw all this traffic to and from ad.something.com and "fixed" it.

0
0
Joke

CDC

They should have outsourced to CDC instead, I hear they have a great remote service.

0
0

@This is why Windows has a bad rep....

Um no, this what happens when you let a moron have admin access. In this case it sounds like the moron was the sysadmin. I don't care what OS you have, admin priv + idiot = disaster

0
0
Gates Horns

@ Vince

"Um, so, they have AD, which natively supports, and is designed for redundancy.

Are you seriously telling me, that someone managed to hose all the AD servers, and all of the GCs? ?!?! ?!?!? What in the name of god are these clowns doing."

Vince, AD isn't designed for redundancy as much as it is designed for replication.

This means that if you delete a large chunk of something from one DC, that deletion gets replicated to all the other domain controllers within a few hours, at most. So, the sequence:

Right-click <part-of-AD>

Select "Delete"

Click "Yes"

Releases the disaster throughout the domain. As for the GCs, well, they get their information from AD, so they're automatically crippled as well.

I suspect they were not using subdomains as they should have on a large network, thus allowing the disaster to be more widespread.

This is a prime example of why to _not_ take the lowest bidder. Take the median bidder.

Netgeek

0
0
Coat

Not a UK company...

@Joe K

Sadly? CSC is not a UK company, they are septics, wasn't too sorry to part company with them myself, best thing ever to happen to me! (you have to hit rock bottom before going up!)

An obvious icon for someone "being requested to leave" ;-)

Oh, and they can do security if they have to, usually with such pedantic stupidity that it's impossible to do any work... the snowdrops know who they are...

0
0
Thumb Down

Re: Saw the same thing in local government once

Anyone with any brains should have recognised that just because the accounts were deleted it didn't mean they were gone from AD. The tombstone period is 60 / 180 days.

You could fix the problem in 5 minutes with ADRestore or Quest Object Restore for Active Directory which are both free tools.

You should have been asked to leave the site for not having a clue either.

http://www.petri.co.il/recovering-deleted-items-active-directory.htm

0
0
Thumb Down

CSC=Cash Sucking Company

What do you expect when you outsource? You outsource to get the 'same job' done for a lower price. The bean counters don't realise (still) that you get what you pay for.

I was outsourced to CSC a few years ago and saw what actually happened. They were willing to pay service penalties to a lesser client so that resources could be moved over to the cash cows.

SInce joining CSC and then eventually taking VR, there have been quite a few re-structuring (redundancy) exercises.

CSC's golden rule was/is, 'If someone in UK leaves, replace with someone 'offshore' (mainly India).

Nothing new to see here. Pleae move on!

0
0
Silver badge
Linux

Outsourcing

Yep, I've seen it all before. I've seen it take 3 days for an outsourcer to get a netware server back because they only had windows admins on-site. Unfortunately the company file & print was netware.

Even the windows admins were inefficient - they had 14 windows admins replacing 7 contractors. The outsource staff were bussed from up north to rental houses down south every week.

If you outsource, your IT department suddenly has different business objectives to the rest of the company which has to be bad.

Windows gets a bad rep because there is a vast pool of "admins" who know how to set up file-sharing and that's about it. Unix has more arcane syntax (it isn't really much more difficult) which filters out those who really shouldn't be administering it. If you know unix, you can and probably would set up redundant systems. The same can't always be said of someone who knows how to stick a windows server cd in the drive.

Tux, because he keeps away the baddies but has a beautiful desktop.

0
0
Anonymous Coward

site visits

"engineer on site sits for 30 mins in reception"

From my experience of visiting one of their datacenters in south east england I'd say that was a little unestimated. If it was planned - you were fine. If it was an emergency you could be there hours just trying to get through security to fix a problem for one of *their* clients.

Quite amusing. You'd have the clients hitting the wall, us desperate to do the tiny easy fix, and yet CSC would have you stuck in reception. I even watched them turn away a BT engineer who had someone waiting on site for them as they weren't scheduled to fix the failure.

Thank god they don't run the emergency services. You have to organise the fire months in advance to ensure a fire engine turned up :)

0
0

Assumptions

What a load of assumptions you people make.

None of you have the true facts, I work fo CSC and if you dont like the company that is fine, however the attacks that you have on the staff are deplorable, CSC have some excellent iT staff, perhaps the stories that you tell sound good.

Or perhaps you are just bitter and twisted people.

How brave you all are, to remain anonymous.

John

0
0
IT Angle

Chicken Shit Company

As per other posters, its no surprise. I was also TUPEd over to CSC from NGC in 04 and could immediately see where it was going. My first clue came from my due diligance meeting with them... I think that Crusty the Clown and Sideshow Bob would have listened with interest a peco-second longer than the two CSC Muppets (sorry Muppets. No disrespect intended) did. I left (VR) a few years ago and still have friends on the inside who have now told me that the critical operational infrastructure is now being in-sourced again (so it could have been worse). Doesn't really show that much confidence in your provider does it?

One quick reply to Mr Crowe. The best staff are the TUPEd staff who know the infrastructure (especially in Grids case). When you lose these key players your in a world of hurt and no amount of hand-over can pass on 10 or 20 years of experience. Remember, its the Management to blame not the grunt on the ground.

And YES, you bet your arse I'm bitter. Its sad to see a well oiled working IT infrastructure go down the pan due to a few bean counters on both sides of the fence/pond.

0
0
Anonymous Coward

Outsourcing to get the best (lowest) price

A few years ago I had the thought that the financial (bean counter) types who fuel the decisions about outsourcing should be required to apply the same algorithms (go for the lowest price regardless) to their own spending (company car).

This would help them live the bargain basement dream / nightmare. No more top of the range beamers, and instead the bottom of the range Ladas and Kias with vinyl seats.

0
0
Paris Hilton

Computer Service Cowboys

Having worked on the NG account for CSC this doesn't surprise me, I was tasked to carry out some work on the application servers and was given full enterprise admin rights because it was *quicker* than sorting out proper delegated admin.

One thing not emphasised enough by the spokesperson is that the file/email servers are on a separate air-gapped network to the mission critical systems so this failure wouldn't have had an effect on anything to do with energy distribution.

John Crowe: Whilst I agree that there are some decent IT staff, CSC, in it's short sighted approach to keeping the shareholders sweet, has failed to realise the long term impact of letting most of it's best IT staff go via Voluntary Redundancy and replacing their local site knowledge with anonymous offshore teams.

Not that I can complain too much about VR though, free cash for not having to work there anymore - brilliant!

0
0
Alert

Outsourcing : Please give me one good example?

Man, there's gotta be one good outsourcing experience out there? I've seen it go outsource, back to in-house and then back to outsource and the amount of skills/logic/time lost to the exercise has left nothing but bitterness about the whole thing

I'd love to hear of one where service was actually improved and made more cost-effective in the prcoess

I'm just confused as to why so many companies see this as a "Golden Solution"

0
0
Alert

This is the same CSC that was...

Brought in to rescue the joke of the Health Service's National Project for IT and we all know how well that is doing!

0
0
Stop

@ John Crowe

Sorry, but I am unable to find you in the CSC directory.

AC, because I don't want to be part of this big RIF round.

0
0

How easy it is to criticise

How easy it is to criticise, especially from behind a screen.

It is correct to have coward in the titlr 'anonymous coward'

Savoy6, you tell me how the Health Service's National Project for IT is going?

Are you involved or do you believe all that you read in the papers.

Well I am involved with NHS work and I actually know how it is going.

Yes anonymous coward, (Chicken Shit Company) you cant beat experience, but remember not all the NG staff who were Tuped over were good and some are actually still working on the account.

John

0
0

Likely to be seperate lans.

I'd have expected, and was confirmed above by someone that the grid control system and office networks are seperate networks, as is typical of pretty much every powerstation in the UK, it certainly is at the one I work in. If it wasn't - i'd be worried that they sit users on the same network as the control system. There is also a whole other DR site dedicated to controlling the grid, which if they both shared a central infrastructure would kinda defeat the purpose of DR.....

I've thankfully never experienced an outsourced infrastructure, but for friends who have it's generally a horror story.

Went to NGC HQ once, 'mission control' (real name unknown but its the grid managmeent room) is a cool place, very James Bond esque :)

0
0

This post has been deleted by a moderator

IT Angle

CSC

The key problem is CSC, they promise low prices etc to get in the door, once in they charge for everything and cut back everything they can and some that they should not.

I would never work for these shower of clowns ever again, in fact, the harder you worked, the less reward you were given, in fact you were actually likely to be made redundant (of course by email without warning).

0
0
This topic is closed for new posts.