Sign in / up

back to article National Grid computers locked-down in outage cock-up

National Grid suffered a major computer system outage last week, leaving many of its staff twiddling their fingers while services supplier Computer Sciences Corporation (CSC) scurried to provide a fix. The electricity and gas operator, which supplies energy to homes and businesses throughout England and Wales, confirmed the …

COMMENTS

House rules Send corrections
This topic is closed for new posts.
  1. Steven Hewittt
    Stop

    This is why Windows has a bad rep....

    twats like these.

    How on earth can you 'delete AD'.....? Surely there could only be a couple of people who have access to the actual database file itself. Are you telling me that they deleted the database from the cluster?!?!? WTF were they thinking!?!?!?!

    Alternativley some pissed off employee decided to delete all the objects in the directory... which makes you wonder if they have heard of delegation or do they just give all helpdesk staff domain admin rights....?!

    And how the fuck can it take more than a couple of hours to restore the database...? And as for email, all you should need to do is reboot your Exchange cluster once AD is back up and voila.....

    Jesus - and companies pay these people millions to look after mission critical IT. You couldn't pay me trust them with a fucking toaster.

    0 0
  2. Chris Ellis

    And we're gonna trust them with a national ID card database?

    Hmm....

    0 0
  3. Joe K
    Alert

    This is what you get from Outsourcing, even to UK outsource companies

    You get what you pay for. Cheap=shit.

    How on earth could anyone but the *very* few Enterprise Admins, who should have the knowledge of God himself, be able to do such a thing.

    And where was the backup and emergency plans and procedures. Probably lost in the mess of outsourced dept muppets, contacting outsourced support desks, who refer them to "my mentor", who ask them to log a support ticket before they'll lift a finger.

    Its bloody terrifying.

    0 0
  4. Anonymous Coward
    Thumb Up

    CSC

    They really are that bad, having to work with them on a regular basis it's taken them nearly a year to approve copying some files onto a server. No matter how bad you fuck up you will never look as bad as them...

    0 0
  5. Anonymous Coward
    Stop

    Where have I...

    heard this before?

    Oh, yeah a similar thing happened to us at Network Rail over the weekend. It wasn't all of AD that went missing, just some of the tabs...

    0 0
  6. Peter Leech Silver badge

    Right...

    Um, yeah. I have to agree. Its not exactly that easy to completely kill AD, especially by accident.

    Even IF you did have someone deliberately kill the system surely it doesn't take 24 hours to restore from your previous days backup tape...?

    More importantly, are the same people responsible for the systems that are managing the power grid?

    0 0
  7. Anonymous Coward
    Anonymous Coward

    To Osama...

    ...don't bother, we're doing a great job on our own.

    0 0
  8. Dr. Mouse

    Not surprised

    I was a contractor at NG doing some CAD work a few years ago at their HQ. Put quite simply, CSC didn't know their arses from their elbows.

    I was only there a couple of months, but it seemed to me that the vast majority of the IT bods had just been on one of these "earn more by working in IT" training courses, and thought they knew everything. And the helpdesk bods were worse than those at Orange broadband (and that's saying something!!!)

    In the end I told my boss I wasn't gonna put up with their ineptitude and forced them to pay me extra to use my own laptop for the job. Even though I had to transfer files on floppies, and the laptop was old and on it's last legs, it was quicker than waiting for the ***kwits to get anything done.

    0 0
  9. Anonymous Coward
    Anonymous Coward

    Not a great surprise

    Having worked for CSC as a PM,on implementations and at NGT, they are very quick to give out "cloned rights"

    The concept of admin rights is always seen as "needed to get the job done quicker" rather than a reduced delegated rights model of least privilege.

    The open door "complain about security and get pushed out of it" is an excellent idea for bean counters.

    I had auto forward emails still being sent to a hotmail account nearly a year after leaving, makes you wonder doesn't it.

    0 0
  10. Anonymous Coward
    Anonymous Coward

    Break it down - worst case.

    Ok lets think about this (therory)

    server dies (fire etc)

    incident raised 30 mins

    engineers travel to site 2 hours (heavy snow in solihul)

    rebuild of server cluster 4-6hours

    setup of temp FR inc certificates 30 mins

    install restore agent and restore database 1-2 hours

    send out a tannoy (lets pretend the AD was D/R across the infastructure at different sites) 15 mins

    reboot every infastructure piece of kit 2 hours

    so thats your day gone!

    more likely:

    time to raise incident 1 hour

    call passed to wrong team 29 mins

    call passed to wrong team 29mins

    call passed to right team who smile and go for lunch 30 mins

    team calls the account manager who rubs his hands together 5 mins

    engineer goes "phaaaa its a big job" and drives to site 20 mins

    engineer on site sits for 30 mins in reception

    engineer gets to server room, boots his laptop from a pendrive as the AD master server, restore the AD database to the AD cluster, promotes cluster to master and shutsdown laptop calls account manager 1 hour

    2-3 hours later the account manager get back from his golf lunch and tells the customer the good news - 1 day and sells them a uk wide resilient DR solution that they already had.

    0 0
  11. Vince
    Unhappy

    How did this happen...

    Um, so, they have AD, which natively supports, and is designed for redundancy.

    Are you seriously telling me, that someone managed to hose all the AD servers, and all of the GCs? ?!?! ?!?!? What in the name of god are these clowns doing.

    0 0
  12. Thaddeus Aid

    you pay millions to set up the network

    and minimum wage for the admin staff and this is what you get.

    hire some qualified admins...

    0 0
  13. Anonymous Coward
    Coat

    24 hours...

    Did it take them 24 hours to restore the backup or did it take them 23 hours of that to work out how to log back in and get their admin rights back so they could restore the backup?

    Wonder if they had to physically take the backup take and the admin to the machine. Outsourcing, you've got to love it.

    0 0
  14. Anonymous Coward
    Paris Hilton

    CSC Clowns.

    Why didn't they restore from backup? Because they didn't have one. Nor is this the first time something like this has happened and the CSC cretins have had to admit their backups haven't been running.

    Frankly I wouldn't let them near my home PC let alone anything else.

    Paris, because she'd almost certainly do a better job.

    0 0
  15. Anonymous Coward
    IT Angle

    Saw the same thing in local government once

    Different company, but the scenario was as follows:

    Onsite AD admin, bloke barely knew his way round a windows desktop, his career to that point had been Novell - He's on the blower to a 'trusted service partner' (Guy at 'large PC builder' doing support while playing solitiare)

    "So whats the easiest way to reset permissions on the desktop accounts?"

    "Uh-huh so after I delete them?"

    "Backup? How do I restore from that"

    Cradles phone for a second...

    "Any idea where the backup for the AD box is?"

    Cue silence....

    Swiftly followed by the account manager going very pale as he realises the penalty fees for loss of service, which accumulate hourly.

    "So how do we recreate accounts for all the machines in the schools across the city?" he asks

    Me and my cohort look him in the eye and come up with

    "Well, if you had the computer accounts in a spreadsheet or the like we could try and knock up a script, but you'll still need onsite to rejoin each machine to the domain from their side"

    "But it's a bank holiday... and we don't even have that list"

    I was asked to leave the site for not being a team player - I said no when they asked me to drive round 20 odd schools on a bank holiday weekend.

    Anon as anyone who was onsite that day will recognise it immediately

    0 0
  16. Anonymous Coward
    Anonymous Coward

    Expected from a privatised state utility

    The National Grid company no doubt has a total monopoly over the electricity wires in this country (or at least the big-uns). I do not think that utilities should be run as for-profit enterprises, because ultimately the business will be run for the owners rather than society it is meant to service. Obviously when the state run these utilities they become lumbering great money pits, but when private enterprise runs the utilities the utility becomes a money spinner for a few, then require lumbering great state regulators aswell! I don't know how utilities should be run... maybe as some kind of non-profit organisation?

    Basically, NGC would have got their IT services from the cheapest bidder: CSC. It is no wonder that NGC end up having a major outage because they have contracted in clowns [1]. They go for the cheapest bidder because the people making the financial decisions don't have a clue about technology, but do have a clue about their bonus if they hit their targets (stuff like implementing supported computers for cheap).

    [1] The only difference between any of these IT consultancy companies is how well the sales staff can lie to the prospective client: In the end, the consultants will charge more after the contract has been signed for services that the client needed and was never sold, or services will be sold to them (as mentioned above).

    0 0
  17. Eugene Goodrich

    The scary bit...

    ... is that if this is what it looks like when you're safe, imagine what it looks like when you're in trouble.

    0 0
  18. Mike Bronze badge

    @You couldn't pay me trust them with a fucking toaster.

    actually, you pay them (indirectly) to keep your toaster running!

    or not running as the case may be...

    0 0
  19. Daniel B.

    Oh god...

    "One source claimed that Active Directory (AD) clusters on the energy giant’s Windows 2003-based server were deleted late last Tuesday. "

    So they're using Windows there. I hope they're not using it to actually *run* the power grid!

    Then again, back in 1993 one hydro powerplant went offline here in Mexico, taking down power in *nine states*. Why did it go offline? Lightning struck one critical piece of equipment. Oops!!!

    0 0
  20. Wayland Sothcott
    Flame

    computers are crap anyway

    Once the user has all their stuff on it then they are yours to bill as much as they will accept. It's only word documents and email, nothing important.... Oh wait wrong thread, it's the UK's Electricity supply, if that goes off, all computers crash!!!

    0 0
  21. heystoopid
    Alien

    Sadly

    Sadly , we were warned about this ever increasing style of management incompetence back in 1969 !

    0 0
  22. Snert Lee

    Probably...

    Baby firewall admin saw all this traffic to and from ad.something.com and "fixed" it.

    0 0
  23. Patrick O'Reilly
    Joke

    CDC

    They should have outsourced to CDC instead, I hear they have a great remote service.

    0 0
  24. kain preacher

    @This is why Windows has a bad rep....

    Um no, this what happens when you let a moron have admin access. In this case it sounds like the moron was the sysadmin. I don't care what OS you have, admin priv + idiot = disaster

    0 0
  25. Anonymous Coward
    Gates Horns

    @ Vince

    "Um, so, they have AD, which natively supports, and is designed for redundancy.

    Are you seriously telling me, that someone managed to hose all the AD servers, and all of the GCs? ?!?! ?!?!? What in the name of god are these clowns doing."

    Vince, AD isn't designed for redundancy as much as it is designed for replication.

    This means that if you delete a large chunk of something from one DC, that deletion gets replicated to all the other domain controllers within a few hours, at most. So, the sequence:

    Right-click <part-of-AD>

    Select "Delete"

    Click "Yes"

    Releases the disaster throughout the domain. As for the GCs, well, they get their information from AD, so they're automatically crippled as well.

    I suspect they were not using subdomains as they should have on a large network, thus allowing the disaster to be more widespread.

    This is a prime example of why to _not_ take the lowest bidder. Take the median bidder.

    Netgeek

    0 0
  26. Anonymous Coward
    Coat

    Not a UK company...

    @Joe K

    Sadly? CSC is not a UK company, they are septics, wasn't too sorry to part company with them myself, best thing ever to happen to me! (you have to hit rock bottom before going up!)

    An obvious icon for someone "being requested to leave" ;-)

    Oh, and they can do security if they have to, usually with such pedantic stupidity that it's impossible to do any work... the snowdrops know who they are...

    0 0
  27. Anonymous Coward
    Thumb Down

    Re: Saw the same thing in local government once

    Anyone with any brains should have recognised that just because the accounts were deleted it didn't mean they were gone from AD. The tombstone period is 60 / 180 days.

    You could fix the problem in 5 minutes with ADRestore or Quest Object Restore for Active Directory which are both free tools.

    You should have been asked to leave the site for not having a clue either.

    http://www.petri.co.il/recovering-deleted-items-active-directory.htm

    0 0
  28. Anonymous Coward
    Thumb Down

    CSC=Cash Sucking Company

    What do you expect when you outsource? You outsource to get the 'same job' done for a lower price. The bean counters don't realise (still) that you get what you pay for.

    I was outsourced to CSC a few years ago and saw what actually happened. They were willing to pay service penalties to a lesser client so that resources could be moved over to the cash cows.

    SInce joining CSC and then eventually taking VR, there have been quite a few re-structuring (redundancy) exercises.

    CSC's golden rule was/is, 'If someone in UK leaves, replace with someone 'offshore' (mainly India).

    Nothing new to see here. Pleae move on!

    0 0
  29. P. Lee
    Linux

    Outsourcing

    Yep, I've seen it all before. I've seen it take 3 days for an outsourcer to get a netware server back because they only had windows admins on-site. Unfortunately the company file & print was netware.

    Even the windows admins were inefficient - they had 14 windows admins replacing 7 contractors. The outsource staff were bussed from up north to rental houses down south every week.

    If you outsource, your IT department suddenly has different business objectives to the rest of the company which has to be bad.

    Windows gets a bad rep because there is a vast pool of "admins" who know how to set up file-sharing and that's about it. Unix has more arcane syntax (it isn't really much more difficult) which filters out those who really shouldn't be administering it. If you know unix, you can and probably would set up redundant systems. The same can't always be said of someone who knows how to stick a windows server cd in the drive.

    Tux, because he keeps away the baddies but has a beautiful desktop.

    0 0
  30. Anonymous Coward
    Anonymous Coward

    site visits

    "engineer on site sits for 30 mins in reception"

    From my experience of visiting one of their datacenters in south east england I'd say that was a little unestimated. If it was planned - you were fine. If it was an emergency you could be there hours just trying to get through security to fix a problem for one of *their* clients.

    Quite amusing. You'd have the clients hitting the wall, us desperate to do the tiny easy fix, and yet CSC would have you stuck in reception. I even watched them turn away a BT engineer who had someone waiting on site for them as they weren't scheduled to fix the failure.

    Thank god they don't run the emergency services. You have to organise the fire months in advance to ensure a fire engine turned up :)

    0 0
  31. John Crowe

    Assumptions

    What a load of assumptions you people make.

    None of you have the true facts, I work fo CSC and if you dont like the company that is fine, however the attacks that you have on the staff are deplorable, CSC have some excellent iT staff, perhaps the stories that you tell sound good.

    Or perhaps you are just bitter and twisted people.

    How brave you all are, to remain anonymous.

    John

    0 0
  32. Anonymous Coward
    IT Angle

    Chicken Shit Company

    As per other posters, its no surprise. I was also TUPEd over to CSC from NGC in 04 and could immediately see where it was going. My first clue came from my due diligance meeting with them... I think that Crusty the Clown and Sideshow Bob would have listened with interest a peco-second longer than the two CSC Muppets (sorry Muppets. No disrespect intended) did. I left (VR) a few years ago and still have friends on the inside who have now told me that the critical operational infrastructure is now being in-sourced again (so it could have been worse). Doesn't really show that much confidence in your provider does it?

    One quick reply to Mr Crowe. The best staff are the TUPEd staff who know the infrastructure (especially in Grids case). When you lose these key players your in a world of hurt and no amount of hand-over can pass on 10 or 20 years of experience. Remember, its the Management to blame not the grunt on the ground.

    And YES, you bet your arse I'm bitter. Its sad to see a well oiled working IT infrastructure go down the pan due to a few bean counters on both sides of the fence/pond.

    0 0
  33. Anonymous Coward
    Anonymous Coward

    Outsourcing to get the best (lowest) price

    A few years ago I had the thought that the financial (bean counter) types who fuel the decisions about outsourcing should be required to apply the same algorithms (go for the lowest price regardless) to their own spending (company car).

    This would help them live the bargain basement dream / nightmare. No more top of the range beamers, and instead the bottom of the range Ladas and Kias with vinyl seats.

    0 0
  34. Mark
    Paris Hilton

    Computer Service Cowboys

    Having worked on the NG account for CSC this doesn't surprise me, I was tasked to carry out some work on the application servers and was given full enterprise admin rights because it was *quicker* than sorting out proper delegated admin.

    One thing not emphasised enough by the spokesperson is that the file/email servers are on a separate air-gapped network to the mission critical systems so this failure wouldn't have had an effect on anything to do with energy distribution.

    John Crowe: Whilst I agree that there are some decent IT staff, CSC, in it's short sighted approach to keeping the shareholders sweet, has failed to realise the long term impact of letting most of it's best IT staff go via Voluntary Redundancy and replacing their local site knowledge with anonymous offshore teams.

    Not that I can complain too much about VR though, free cash for not having to work there anymore - brilliant!

    0 0
  35. Tim Hustler
    Alert

    Outsourcing : Please give me one good example?

    Man, there's gotta be one good outsourcing experience out there? I've seen it go outsource, back to in-house and then back to outsource and the amount of skills/logic/time lost to the exercise has left nothing but bitterness about the whole thing

    I'd love to hear of one where service was actually improved and made more cost-effective in the prcoess

    I'm just confused as to why so many companies see this as a "Golden Solution"

    0 0
  36. Savoy6
    Alert

    This is the same CSC that was...

    Brought in to rescue the joke of the Health Service's National Project for IT and we all know how well that is doing!

    0 0
  37. Anonymous Coward
    Stop

    @ John Crowe

    Sorry, but I am unable to find you in the CSC directory.

    AC, because I don't want to be part of this big RIF round.

    0 0
  38. John Stoddard

    How easy it is to criticise

    How easy it is to criticise, especially from behind a screen.

    It is correct to have coward in the titlr 'anonymous coward'

    Savoy6, you tell me how the Health Service's National Project for IT is going?

    Are you involved or do you believe all that you read in the papers.

    Well I am involved with NHS work and I actually know how it is going.

    Yes anonymous coward, (Chicken Shit Company) you cant beat experience, but remember not all the NG staff who were Tuped over were good and some are actually still working on the account.

    John

    0 0
  39. Paul Stephenson

    Likely to be seperate lans.

    I'd have expected, and was confirmed above by someone that the grid control system and office networks are seperate networks, as is typical of pretty much every powerstation in the UK, it certainly is at the one I work in. If it wasn't - i'd be worried that they sit users on the same network as the control system. There is also a whole other DR site dedicated to controlling the grid, which if they both shared a central infrastructure would kinda defeat the purpose of DR.....

    I've thankfully never experienced an outsourced infrastructure, but for friends who have it's generally a horror story.

    Went to NGC HQ once, 'mission control' (real name unknown but its the grid managmeent room) is a cool place, very James Bond esque :)

    0 0
  40. Anonymous Coward
    IT Angle

    CSC

    The key problem is CSC, they promise low prices etc to get in the door, once in they charge for everything and cut back everything they can and some that they should not.

    I would never work for these shower of clowns ever again, in fact, the harder you worked, the less reward you were given, in fact you were actually likely to be made redundant (of course by email without warning).

    0 0
This topic is closed for new posts.