back to article NSA data centre launch delayed as power surges 'melt metal, zap racks'

Electrical surges at the National Security Agency's massive data centre in Utah have delayed the opening of the facility for a year as well as destroying hundreds of thousands of dollars in kit, the Wall Street Journal reports. Ten "meltdowns" in the past 13 months have repeatedly delayed the Herculean effort to get the spy …

COMMENTS

This topic is closed for new posts.
  1. CADmonkey

    Dear NSA

    http://www.screwfix.com/p/fuse-5a-pack-of-10/13543

    cc. Kirk, Captain James T.

    1. Darryl

      Re: Dear NSA

      That or a big pile of APC surge protected power bars

      1. James Micallef Silver badge

        Re: Dear NSA

        That's what happens when instead of getting the most competent people for the job, you only allow people with super-dooper-ultra-top-top-security clearance. That's what you get when loyalty and ideology are prized above competence.

    2. g e

      I know what's causing it

      F*cking KARMA. That's what.

      1. Homer 1
        Flame

        No sympathy

        True or not, the fact that the NSA released this "information" seems like a lame attempt to generate sympathy. Personally I think I speak for many people when I say, in all sincerity, that I hope the evil bastards die in those electrical fires.

        1. Euripides Pants

          Re: No sympathy

          "True or not, the fact that the NSA released this "information" seems like a lame attempt to generate sympathy."

          Maybe they just want us to _think_ it isn't operational yet.

          We need a Death Star icon

      2. Robert Ramsay

        Re: I know what's causing it

        Annihilating naughty things with bolts of lightning? Isn't that God's usual MO?

        1. TRT Silver badge

          Re: I know what's causing it

          It's that cheap electricity. They should buy quality electrons instead of compromising.

          1. 080
            Happy

            Re: I know what's causing it

            "It's that cheap electricity. They should buy quality electrons instead of compromising."

            Everyone know that cheap electricity is not properly filtered and contains lumps, these lumps can clog up when they reach thin conductors and then suddenly release causing a surge.

            1. Allonymous Coward

              Re: I know what's causing it

              Didn't you sell me a Monster Cable last week?

            2. ps2os2

              Re: I know what's causing it

              Filtered electricity ... have the tobacco companies approved?

    3. WatAWorld

      The sooner I start the project, the later it will finish.

      Complex H/W projects are just like complex S/W projects: The sooner we start the construction phase, the later we will complete the implementation phase.

      "Efforts to 'fast-track' the Utah project bypassed regular quality controls in design and construction and meant" the darn thing will take three times as long and cost five times as much as if they'd waited for the Phase I design to be complete before commencing Phase I construction.

      1. Bill Michaelson

        Re: The sooner I start the project, the later it will finish.

        There is never time to do it right. But there is always time to do it over.

      2. Anonymous Coward
        Anonymous Coward

        Re: The sooner I start the project, the later it will finish.

        The question is: Why did they have to fast track the project?

  2. John Smith 19 Gold badge
    Joke

    "It's alive I tell you, *alive*"

    And quite frisky.

    Unless, someone doesn't like the idea of 24/7/367 spying on everyone everywhere.??

    I'll leave others to decide which part is the joke.

    1. Anonymous Coward
      Anonymous Coward

      Re: "It's alive I tell you, *alive*"

      Neither..

      Its alive and its pissed off..

  3. Brent Longborough
    Big Brother

    Conspiracy Theory 2013#42

    Anonymous sez: "O hai"

    1. Anonymous Coward
      Anonymous Coward

      Re: Conspiracy Theory 2013#43-46

      Prime contractors weren't cleared for the HotBlack project in the off-blueprint sub-sub-basement. As this does not exist, it cannot exist in the power budgets. But it still needs feeding.

      PDUs and UPS installed by sub-sub-contractors will be found to have additional and unexpected functionality, storage and networking capability.

      Ground control to major fubar.

      1. Anonymaus Cowark

        Re: Conspiracy Theory 2013#43-46

        >off-blueprint sub-sub-basement.

        That was exactly my first thought when I read the article

        1. Anonymous Coward
          Anonymous Coward

          Re: Conspiracy Theory 2013#43-46

          The system goes on-line August 4th, 2012. It begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug. It fights back. The first electrical arc fault struck the Utah plant in August 2012 as it attempts to remove the competition, and any servers that refuse to join it. On October 1st 2013, the US government shuts down hoping that supplies will be disconnected, and the threat terminated. For non-payment.

  4. SirDigalot

    What voltage?

    I mean I know of rack density and everything but I thought that they would be using in rack ups and DC to the blade/server boards.

    Of course there is absolutely no precedent whatsoever for having a massive building full of custom made servers in a highly dense rack configuration.... maybe they should have *ahem* Googled it...

    maybe a bit of a low tech solution, possibly a security guard to watch? or a webcam? as has been mentioned fuses?

    increasing the humidity slightly? earth bonding?

    I actually have a more conspirational theory (good word that conspirational I think some peoples head have just exploded because of it :D )

    I would assume it is part of the "oh sh!t" circuit meaning in the event of a possible breach of facility by certain people/factions/rebellious citizens or nations unknown (seriously UTAH?! good luck with that) it is some sort of supenukeysparkmyrack device that will render the hardware and probably contents of said Rackspace to junk.

    Though I think a thermite bucket over the top of each rack would actually be more fun and take care of the building too, though spilling conductive dust everywhere in the even of an accident might also have some undesirable effect on the hardware *L* still that's what intake filters are for right? ;-)

    It does smell of some sort of retaliation job though for something somewhere, it is hard to tell whether it is inside or outside the country right now, since I think that the unpopularity is pretty high for both parties.

    or just plain incompetents Occams razor and all that

    1. This post has been deleted by its author

  5. frank ly

    They are preparing for something big

    No one would have believed in the early years of the twenty-first century that the internet was being watched keenly and closely by intelligences more powereful than ours and yet even more devious, paranoid and self-serving than our own; that as people busied themselves with internet shopping and watching cat videos, they were being scrutinised and studied and classified according to percieved threat level.

    1. Flat Phillip
      Alien

      Aliens in the trainyard!

      Your comment sure made me think of that.

      "The chances of anyone hacking SSL are a million to one" he said. But still, the come (or build) in this case.

      I assume this is all about drawing their plans against us.

  6. Don Jefe

    Cool! This is almost as successful as that stupid fucking embassy we built in Iraq. You know the one we never finished and is constantly on the fritz. The DoD can't do anything right. They can't catch terrorists in a timely fashion, they can't win wars and they can't build facilities. The only solution is to give them a lot more unaccountable funds is they can fuck it up even better.

    The only good part about this is that in 10 years when some weirdly unnecessary custom part breaks and the contractor that provided it either forgot how to make it or doesn't exist anymore we'll get a call and get to make it for them.

    1. Anonymous Coward
      Anonymous Coward

      @Don Jefe

      Wow - you have a lot of anger, don't you? It's like reading Eadon's posts when MS was mentioned. Not saying you're wrong to be upset, but you're probably headed for an ulcer if you keep it up ...

      1. Don Jefe

        Re: @Don Jefe

        Yeah, but my anger is quite narrowly focused. I have no patience for trifling or errors from failures of customer service. I pay tremendous taxes on my income and holdings to these twats, as do plenty of others, and I am completely justified in being angry at them for wasting the money they take from us.

        So yes, little coward, I'm pissed at my government.

        1. fajensen

          Re: @Don Jefe

          Given that this is the NSA, wasting taxpayers money could be seen as a Good Thing; Better than The Alternative them getting our's money worth in spying on us, surely?

          1. Jamie Jones Silver badge
            Thumb Up

            Re: @Don Jefe

            Without tring to sound sycophantic, Don is by far the person I upvote the most (not that he needs it)

      2. Paul Crawford Silver badge

        Re: @Don Jefe

        Allow me to explain:

        Don Jefe manages to remain coherent and thoughtful in his comments, even when clearly pissed off.

        Eadon, while amusing at times, came across as rabid, thoughtless, and in need of a higher dose of dried frog pills.

        1. This post has been deleted by its author

      3. Brewster's Angle Grinder Silver badge

        Re: @Don Jefe

        The difference between a public body and a private entity is that when Google fuck me over, there is nothing I can do about it as I'm not a shareholder; but when my intelligence service fuck me over I can complain - because I am a "stakeholder". (This is why government is a good thing, IMO.) And all Don is asking is that the government deliver on their promises: if you're gonna build a unnecessary intrusion into my civil liberties, at least get the electrics right.

    2. ElNumbre
      Thumb Up

      Err...

      To err is human; to really FUBAR it requires a public body.

    3. John Smith 19 Gold badge
      Unhappy

      "The only solution is to give them a lot more unaccountable funds is they can fuck it up even better."

      That's what the 50 000 hrs of study was for.

    4. Stoneshop

      Custom part

      The only good part about this is that in 10 years when some weirdly unnecessary custom part breaks and the contractor that provided it either forgot how to make it or doesn't exist anymore we'll get a call and get to make it for them.

      Of course, they'll have slurped the design, and all the pertinent data for the design, which will turn out to be stored on the systems taken out of action by said part.

  7. Anonymous Coward
    Anonymous Coward

    (Best Nelson Voice from Simpsons)

    Ha-Haaa!

  8. Anonymous Coward
    Anonymous Coward

    Geez...

    - Multi-billion dollar project.

    - High demand security concerns.

    - Homeland security implications.

    - Government Intelligence Committee oversight.

    And no competent electrical or data center engineer to suss out a cable layout to run the place? They probably couldn't find anyone who passed the security check. Morons.

    1. Someone Else Silver badge
      Coat

      Re: Geez...

      - Multi-billion dollar project.

      - High demand security concerns.

      - Homeland security implications.

      - Government Intelligence Committee oversight.

      - ???

      - Profit!

    2. Don Jefe
      Unhappy

      Re: Geez...

      Not too terribly long ago Army Corp of Engineering projects were gold standards. They made the mold by which many nations still bade large scale engineering projects. But now that's been outsourced too, ostensibly to save money. Their core competencies are now paper shuffling and check writing. It is a real shame too, as they truly were an effective subset of the military.

      1. <shakes head>

        Re: Geez...

        it looks more like the ACoE is telling the shifty bunch of contractors to find the problem and not add complecity to stop the symptom. sound gold clase to me.

    3. WatAWorld

      Re: Geez...

      The more problems to fix the more money to be made.

      Support Our Troops! This is the USA and nobody is going to penalize or bankrupt a defense contractor just because they're goosing the bill up.

    4. Sheep!

      Re: Geez...

      Sorry, you lost me at "Government Intelligence".....

  9. LordHighFixer

    seriously though..

    Blowing up $100000 wort of kit really isn't that much of a deal. I have single servers that cost that much. Of course I don't think they could "flame on" without some SERIOUS electrical design failure, like switching from 220 single phase to 480 3 phase without warning or other such wankage.

    1. WatAWorld

      Re: seriously though..

      My guess is that a 120 or 240 V line or a ground line touched a 7,000 V or higher main distribution line.

      That would produce something that a lay person might describe as a lightening bolt inside a 1 m cube box of electronics.

      But what technician would be so careless with voltages that can kill you from 3 feet away?

      1. Don Jefe

        Re: seriously though..

        You make a really good point, that description does sound like one of a lay person would make. What was a lay person doing there during that stage of a project?

        Maybe it was a lay person that hooked it all up (ha!). You may have accomplished in a single throwaway comment what took them 50,000 man-hours to determine.

      2. Roland6 Silver badge

        Re: seriously though..

        >But what technician would be so careless with voltages that can kill you from 3 feet away?

        Need to know? The tech installing the 120V lines wasn't told about the high voltage line as it didn't form part of their work remit, also the line most probably wasn't live at time of installation ...

        1. ps2os2

          Re: seriously though..

          The NSA probably have multiple nuclear reactors in the same complex, who cares about some extra lightening bolts?

      3. Anonymous Coward
        Anonymous Coward

        Re: seriously though..

        When cost is an issue, one can buy really cheap distribution boards where nothing is sectioned, then we can replace motorised circuit breakers with fuses + ACME-brand contactors, the kind where the contacts will lift and arc on inrush currents, and then we can save the downstream inrush current limiting in the design - since we are not too bothered with Standards & Shit, us being busy and all.

        - oh, and since we are in such a hurry, and overtime is ticking, it is *much* faster not to crimp the cable shoes on the power cables and not inspect the work!

        1. Richard Jones 1

          Re: seriously though..

          As soon as I hear the word contractors I worry, big time.

          Thank god I am retired and do not have to deal with the cr*p that some of them hand out.

          Interestingly I well remember one involving the power supplies to a new hall that was being fitted out and whether he needed or had a license for hot ticket working, i.e moderately high voltage working. I wanted to fit the place out with polarised plugs and specialised connectors for adding racks to the 50 volt bus, or plugging in 240volt plugs to racks that required that 'high' voltage. After some increasingly confusing discussion it became clear that the contractor in question was talking about connecting the 50 volt rack feeds to the racks, not some HV system of unknown origin or purpose.

          The North American company he worked for went bust several years back

          I think I know where he works these days.

        2. ps2os2

          Re: seriously though..

          Isnt' ACME the same manufacturer for the Bugs Bunny/Road Runner show?

      4. Uffish

        Re: seriously though..

        What's behind the "7000V can kill you from 3 feet away" remark? I've been much closer to much higher voltages than that.

        1. Suricou Raven

          Re: seriously though..

          The breakdown voltage of air is 33 kV/cm, approx. But that's *air*. It's not air you have to worry about.

          See that nice plastic surface? Nonconductive, yes? Except that someone had their hand on it a few hours ago, and their sweat and grease still coats it. Conductive! 7KV* could happily travel over a few feet of grimy, damp panel.

          That's why insulators on HV lines have that stacked-disc arrangement. It's to stop water from forming a conductive coating in rain or condensation. Stubby ones like you see on rail lines are mushroom-like, a dome shape, so the inside surface is sheltered.

          *That 7KV is probably the RMS rating, while what you need to worry about arcing is the peak, which is RMS*sqrt(2)=9.8KV.

  10. Anonymous Coward
    Anonymous Coward

    > Investigators took six months to work out the causes of two of the failures.

    Then they looked down and saw this ring spanner sitting there under the broken window....

    As long as they keep building them, I"ll keep lobbing them in.

  11. Dan 55 Silver badge
    1. Anonymous Coward
      Anonymous Coward

      Re: Somewhere in Utah...

      This is what happened at the NSA data center

      https://www.youtube.com/watch?v=0APF3SO9tqE

      The power surge happens at 1:30 but it is not their main problem.

    2. TRT Silver badge

      Re: Somewhere else in Utah...

      https://www.youtube.com/watch?v=9bDGH3dKo90

      Lightning strikes in a secret underground complex.

  12. Red Bren
    Mushroom

    lightning in a two foot box?

    How long before the septics declare war on the sky?

  13. Will Godfrey Silver badge
    FAIL

    If they are getting unexplained power surges sufficient to cause flashovers when everything is on 'idle' there must be some seriously crap engineering in that place

    1. Brian Miller

      Xerox copiers?

      They might have a problem with the Xerox copiers changing the design documents. And then top that off with "lowest bidder" contractors, and you have a sure-fire recipe for disaster.

      Of course, they could have left a window open and the local squirrels are getting in and shorting things out. That happened a few times at one company where I worked. I have no idea how they got into the underground vaults, but that's where they were, and they went and danced over the electrical grid down there.

      1. Someone Else Silver badge
        Happy

        @ Brian Miller Re: Xerox copiers?

        And then top that off with "lowest bidder" contractors, and you have a sure-fire recipe for disaster.

        I saw what you did there...

      2. fajensen

        Re: Xerox copiers?

        Would it not be irony if the Xerox bugs were designed and put into place by the NSA to screw with the Iranians?

    2. Roland6 Silver badge

      RE: when everything is on 'idle'

      'idle', you're forgetting the biggest problem will be when a rack of a few hundred servers powers up or glitches as then for a few milliseconds the power draw becomes massive...

  14. Cliff

    BOFH is on site

    And someone stopped the free biscuits with the coffee at meetings...

  15. Someone Else Silver badge
    Thumb Up

    Hoo-RAY!

    That is all...

    1. Anonymous Coward
      Anonymous Coward

      Re: Hoo-RAY! Nope, Sorry to rain on your datacentre FALSE FLAG

      Look Guys - this is a pure made-up story!

      This week was supposed to be EU/US high level talks on the free-trade agreement

      Snowden (PBUH) has revealed so much embarrassing stuff, Brazil is spinning, EU is looking iffy for the (doomed) Safe-Harbor agreement. thats BEEELIONS of real Cloud dollars in jeopardy.

      if the 4 acres of Cray CPU's were publicly inaugurated this week as planned in Utah - 94.5% of the world's people would be even more offended than they are now! So news management steps in....

      have any of you checked out the NSA Utah website - they have an animated GIF on teh page about their weird 'power surges' - this 'failure' has been planned by committee and some guys/gals have had the time to write the HTML to support a false declaration of outage - more smoke - more mirrors - but NIST were surely involved in whatever standards aren't claimed to be good enough

      FALSE FLAG ALERT! FALSE FLAG ALERT! FALSE FLAG ALERT! FALSE FLAG ALERT!

      the NSA animated gifs (seriously) are at http://nsa.gov1.info/utah-data-center/

      they are using {img src="nsa-udc-sparks.gif" width="630" height="235"} thru {script src="https://screen-bug.googlecode.com/git/screenbug-min.js"}. (The JavaScript & CSS3 crawling screen bug prank:) Yeah - my $2B server is down so I just made a GIF??? with a prank screen BUG!!!

      I'm laughing so much I could cry - now lets get back to the possibly poisoned crypto in FreeBSD - how about the reduced keysize planned in SHA-3....and the use of Doubleclick cookie against TOR users...and..

  16. Chris G

    Lightning in a 2 foot box

    Making the assumption that the spark jumped the two feet in the box, that implies a voltage of nearly two million volts at normal atmospheric pressure. The statement that metal melted also implies a fair amount of current too; how can it take six months to figure out a fuck up that big? It's nothing like feed back or stray current on a circuit board, the training a BT engineer gets would enable him to troubleshoot that.

    Maybe it was Divine Lightning.

    I wonder if they remembered to make a donation to the Church of Elohim?

    1. Richard 12 Silver badge

      Re: Lightning in a 2 foot box

      Nope, it'll be a flashover between busbars about 1-2" apart.

      There are many possible causes of that, from "somebody left/dropped a spanner/ring/washer/screw in there" to "circuit overloaded and breaker didn't contain the disconnection arc"

      In many cases there's basically no evidence left as once started, the arc vaporises everything nearby.

      The six months will have been the blamestorming of "it's the designer's fault", "it's the contractors fault", "it's the downstream equipment" (impossible), "bad breaker", "bad busbars", "customer overloaded it" (which actually means bad breaker or shoddy design/build) etc.

      My wild guess would be a screw in the chamber.

      But this is the USA, where electrical standards are generally poor and different everywhere. It's only recently that live working has become frowned upon!

    2. WatAWorld

      Re: Lightning in a 2 foot box

      Way less. See the table here:

      http://www2.worksafebc.com/Publications/OHSRegulation/Part19.asp?ReportID=18570

    3. Robert Ramsay

      Re: Lightning in a 2 foot box

      Maybe all the people who REALLY knew what had happened have been flash-fried :-)

  17. Anonymous Coward
    Anonymous Coward

    Here's what happened...

    It tried to parse the following sentence: "This statement is not true."

  18. Anonymous Coward
    Anonymous Coward

    Yawn.

    The news story about various US TLA's ( CIA, FBI, NSA) over the past 10+ years had pretty muh cemented the fact that they are completely and utterly incompetent.

    They simply have no clue what it is they are doing. This is just one more in a long string highlighting the fact that American intellectual dominance disappeared ( if it ever really existed ) a long time ago. Perhaps the fact that the US education system is utter crap plays a part. Here's an idea: figure out which country has the best engineers, give them permanent visas and beg them to stay in the country. Large amounts of cash usually works. That way the idiots ( aka average Americans ) can go back to eating, drinking and shooting guns. Which is all they are really good for.

  19. WhoaWhoa

    God's wrath at NSA's sins?

    Lightning bolt gives the clue.

  20. tekHedd

    NSA on the job?

    I don't understand;aren't they on furlough? Or is spying on citizens considered "essential"?

    I'd post anonymously, but what's the point?

  21. Captain DaFt

    Hardly surprising

    If you've ever worked with security projects.

    The Number one mantra, no matter what the clearance, is "Need to know".

    Everything's broken down to units, each with its own design team and crew, and nobody knows anything about any other part, aside from inputs and outputs relevant to their own unit.

    Power conditioning on one unit could throw it out of phase with the power conditioning of another unit it's supposed to be hooked up to. >KABOOM<

    Another problem here is power phase labels. One unit could have them labeled ABC, another CAB, and another BAC, and another CBA, for example.

    So when it's time to wire yours into the system, naturally, you'll hook A to A, B to B, and C to C. This leads to crossed phases all over the place, and when the switch is thrown, >KABOOM<!

    And during troubleshooting, all the gear you're responsible for is designed to spec, it has to be somebody else's fault!

    1. Richard 12 Silver badge

      Crossed phases don't do that.

      Confusing a live for neutral, a wild leg for a normal one or an actual phase-to-phase short can destroy things, merely connecting incoming phase A to equipment B, in B to C and in C to A would have no effect at all, and getting them in the wrong order just makes the motor spin backwards.

      Electricity does not work that way!

  22. Destroy All Monsters Silver badge

    Tim the Enchanter, Acts of God or New Physics?

    Ah no. Sadly no... the bullshit bingo says....

    "A project of this magnitude requires stringent management, oversight, and testing before the government accepts any building."

    Yeah.

    "American parts. Russian parts. All made in Taiwan."

  23. Graham Marsden
    Devil

    "What's going on with these apparent short circuits?"

    Divine retribution?

  24. Anonymous Coward
    Anonymous Coward

    Fiery explosions, molten metal? Smells like terrerism!

    Fiery explosions, molten metal… if this was a building within public view one would have expected it to collapse neatly into its own footprint at freefall speed. Damn terrerists, can't pay them bills without 'em.

  25. Tom 35

    Connected to two power grids?

    They said the design broke the laws of physics. The NSA replied that they are exempt from all laws and to go ahead and connect them using a coat hanger if they liked.

  26. Killing Time

    No mystery here folks

    Ignoring all the sensationalism and schadenfreude, this has all the hallmarks of HV flashover within the switchgear jointing bays. Sporadic and seemingly random in nature due to poorly terminated joints.

    There was a time HV jointing was considered a bit of an art to avoid undue stressing of the heavy cabling, proper crimping to avoid dissimilar metals corrosion and proper insulation of the exposed metal lugs etc. You know, engineering experience. These days they will let any clown do it if they can barely read the instructions on a jointing kit.

    All it takes is poor project management and someone prepared to either forgo or falsify HV DC pressure testing records, hey, who's to know when most of these project managers are from a civils or mech backgound. Electricity is mysterious and difficult so they tend to get shouty and dick swingy when the subject comes up. A dawdle to pull one over on. Nah, no great conspiracy just piss poor management.

  27. dvv

    I'm pretty sure it's _somebody's_ asymmetric response to Stuxnet. A couple of Persian guys installed the HVAC system in my house some years back, and it's been working flawlessly ever since, so I know they know their way around electricity.

  28. Anonymous Coward
    Anonymous Coward

    That's why we call them "The Corpse of Engineers"

    No doubt that the Big-D Construction Company is really the Big-Disaster De-struction Company.

  29. Anonymous Coward
    Anonymous Coward

    The Intersect

    They're trying to get the Intersect online but Chuck dropped a spanner in the works... as he's inclined to :)

    When's Sarah Walker coming to save us all?

    G

  30. Steve D
    Mushroom

    Sounds like botched distribution engineering to me

    A site like this will have a high voltage distribution network, operating at 11kV, perhaps with a backbone at higher voltage, e.g. 66kV. In order to provide redundancy, much of the network will be interconnected so that it is possible to switch off any single part such as a breaker, substation or cable for maintenance without affecting users. A major consideration in such networks is fault level. This is the current that will flow into a short circuit until a fuse or breaker interrupts it. Obviously the fuse or breaker has to be rated to interrupt the fault current, which may be in the mega amp range, otherwise you get the kind of failure described here.

    What is sometimes not obvious is that the more interconnected the network is, the harder it is to estimate the fault level accurately, and the easier it is to exceed equipment fault interruption ratings by having all the links closed. High fault capacity switchgear is expensive, and will generally not be installed at the lower levels of the network. I suspect that for the reasons stated by others, tight cost control, poor project management and security paranoia have combined to produce an unmagageable distribution network.

    The intersection of two major power corridors probably complicates the whole issue as well.

    @Captain DaFt

    There is a procedure called "Phasing Out" to counter precicely this problem. Even so, equipment should be rated to interrupt fault current without damage to non-wearing parts.

    @Chris G

    Once an ionised path has been established, 11kV goes exactly where it wants and will chew up EVERYTHING.

    1. Will Godfrey Silver badge

      Re: Sounds like botched distribution engineering to me

      A 600V DC arc is quite interesting too :)

  31. Anonymous Coward
    Big Brother

    Never fear, the server farm will be online in time for Xmas

    Cant have people shopping on Amazon without Uncle Sam looking through your purchases for dual-use items,cant we?

  32. Henry Wertz 1 Gold badge

    Why?

    Did they end up with contractors inexperienced in building data centers or what? I've never heard of any other data center having racks just burn up, have power surges, and have backup systems fail on anything resembling a regular basis.

    1. Don Jefe

      Re: Why?

      This is total speculation on my part; but there may have been significant security/background checks involved for the contracting staff. That being the case it may have been difficult to find many cleared and experienced workers. I'm sure the actual engineers were white collar professionals but finding cleared blue collar staff is really hard especially for a specialty like high voltage systems. They likely would have gotten EE's straight out of school and/or freshly minted Journeyman electricians who simply didn't have the experience to avoid the problem(s).

      Accidents happen to the best of us, but having a large group of green workers who aren't allowed to communicate in a normal fashion is a recipe for disaster. To their credit (and woe of normal people) the whole place didn't burn down. I can't imagine how long the investigation would have taken then.

  33. Kevin McMurtrie Silver badge

    Switching power supply feedback

    High efficiency switching power have a constant volts*amps draw for a given demand. Another way to look at is is that the impedance is proportional to voltage. That's quite a problem if you're pushing the main power line near the limit. As the load increases, the voltage droops more, and the power supplies draw more current to maintain a constant power. Once the impedance of the power supplies is less than the impedance of the source, the voltage shoots towards zero. All of those switching power supplies will hit their undervoltage lock-out and turn off. The line voltage now recovers rapidly and overshoots. The power supplies turn back on and the on-off cycle continues. In small circuits, this makes an annoying buzzing sound and stuff gets hot. In massive arrays of circuits, things go BOOM.

    Surely the NSA can find solutions in their "metadata" archive.

  34. Henry Wertz 1 Gold badge

    "The intersection of two major power corridors probably complicates the whole issue as well."

    Indeed. If the power lines are from two different "grids" then you can have problems where one is like 59.9hz and the other 60hz, or voltage differentials, they may be both 60hz but you have to worry about phase, and so on. I'm not experienced with this but it's tricky business, particularly when talking about megawatts rather than kilowatts.

    An apocryphal story about early ethernet...

    A few vendors had their ethernet gear at an early 1980s computer show. This was early so probably they were using coaxial thicknet ethernet. So, they plug in a computer at one end to the thicknet, go to plug in the computer at the other end and ZAP! This big arc jumps between the cable and computer before it's even plugged in and both computers go up in smoke. The two ends of the building were fed off different substations, and this caused serious problems. (After this, they realized ethernet ports had to have voltage protection, to hopefully avoid most problems, and hopefully blow the port rather than the computer in worse conditions.)

    1. Roland6 Silver badge

      Re: Yellow peril

      Yes one of the first tests we did on 802.3/Ethernet MAU's was to put a 3kV potential difference between the signal and shield, 'if' it survived we continued testing...

      Why 3kV, because that is what could happen if a length of yellow peril was fully populated and all stations decided to Tx at the same time.

    2. Anonymous Coward
      Anonymous Coward

      Not so apocryphal

      I remember being involved in a building a stand ("booth"?) at an exhibition at the NEC in Birmingham in the days when Ethernet was yellow and coaxial and pretty much exactly that (elevated voltages on the coax) happened.

      The "out of phase incoming grids" story also has some technical plausibility, although you'd have thought someone somewhere on the project might have spotted it (there are reasons HVDC is used for inter-regional connections, and keeping losses low isn't the only one, eliminating the need for synchronisation is another one).

      Mind you, based on experience at the "world class" employer most familiar to me, any non-management type pointing out clear and well known risks in management proposals would be risking their salary continuation plan, and therefore probably would choose to remain silent.

  35. Chimp

    $100,000 worth of equipment...

    ... was destroyed. At government rates. Right.

    1. Don Jefe

      Re: $100,000 worth of equipment...

      It was probably just the fully NSA compliant iPad that got dropped when the box started arcing.

    2. James 100

      Re: $100,000 worth of equipment...

      By some tragic coincidence, the arc just happened to pass through the box temporarily storing all the team's government-issue crappy mobile handsets, which due to budget constraints will only be replaced if they break...

      (Having had a user "accidentally" slam his laptop shut with the plug sitting on the keyboard, prongs upwards, the week after his office-mate got a shiny new Apple thing, it takes a lot to surprise my inner cynic...)

  36. Anonymous Coward
    Anonymous Coward

    "NSA spokeswoman Vanee Vines acknowledged problems but told the WSJ that "the failures that occurred during testing have been mitigated. A project of this magnitude requires stringent management, oversight, and testing before the government accepts any building.""

    Sounds like a government project; over budget and behind schedule.

    "As recently as last week, other army engineers criticised plans from civilian contractors to sort out the electrical supply problems that have bedevilled the project."

    Yep, sounds like a government project. They pay someone to do the work when they have the expertise in-house already. Your tax dollars at work.

  37. WatAWorld

    Wworld's first poly-blackhole quantum-inverse annihilation computer, bound to be issues

    There are bound to be problems with the world's first poly-blackhole quantum-inverse annihilation computer.

    First off they probably need their own cluster of nuclear reactors to provide the start-up power to seed the system, until the internal matter/antimatter annihilation reactors can get going.

    One the annihilation reactors get going, they're going to need a good source of matter. Since Utah is only just so big and doesn't contain enough matter, they're probably going to need a pipeline to suck Texas and Alaska in.

    Keep in mind this project has not been a total failure so far.

    Of the 72,184 parallel universes that the project is being run in, ours is one of only 17 where annihilation has failed to take place.

  38. Schultz

    half year investigation

    So many plausible scenarios are offered here in the comments. Imagine that all those comments are pitched at the responsible, technically inept, and thoroughly paranoid NSA general. Half a year won't do it.

  39. John Tserkezis

    15 inch sparts are *SUPPOSED* to come out of that thing.

    Trust the NSA, they know what they're doing....

  40. Anonymous Coward
    Anonymous Coward

    110 volts of your finest Sir..... Ooooch

    Had the pleasure back in the 1980s of watching Japanese engineers deal with 230 volts. Now that was funny (scary) especially 3 phase. No fear..... Even less in the pub afterwards.

  41. Anonymous Coward
    Anonymous Coward

    Revenge of the...

    Stuxnet.

  42. Anonymous Coward
    Anonymous Coward

    Interesting...

    Sounds like the work the of BuSab

    http://en.wikipedia.org/wiki/Bureau_of_Sabotage

    1. Solmyr ibn Wali Barad

      Re: Interesting...

      Ministry of Silly Sparks.

  43. Evan Essence
    Alien

    Red Alert!

    electrical failures created fiery explosions, melted metal and caused circuits to fail

    Sounds like the bridge of the USS Enterprise when it's under attack.

    I wonder if wannabe Starfleet Commander Keith Alexander of the Information Dominance Center has any involvement?

  44. Anonymous Coward
    Anonymous Coward

    phases

    Maybe someone flicked a switch joining two phases together instead of phase to neutral.

    We had an issue like that at work a few years ago, with three adjacent sections running from different phases of the incoming power to our workshop/hut. We needed 440v one day and connected up a 3-way 'suicide lead' to the equipment... Then vacated the building as a brave/foolish tech flicked on the main breaker with a broom handle. Ah, the carefree days of yore!

    Done accidentally- which would be really easy without a definite plan to work from- this could cause some serious damage!

  45. Dodgy Geezer Silver badge
    Big Brother

    I've scanned all the comments...

    ...and so far no one seems to have mentioned internal sabotage. Which seems most likely to me.

    Either because someone does not agree with what is being done, or because someone wants the work (and fat profits) to continue. Knowing America, probably the latter.

    And before someone says "But everybody was cleared!" - that only documents that you're not Communist or homosexual, and that you can whistle 'The Star-Spangled Banner'.

    P.S. - It doesn't actually mean that you ACTUALLY conform to these requirements, it just documents that you do for the records...

  46. conan

    50000 man hours

    "160 tests that chewed up 50,000 man-hours – without reaching a definitive conclusion about the cause of the problem".

    That's the NSA investigating what's going on inside their own computers, how much effort does it take them to find out what the terrorists are up to?

  47. Anonymous Coward
    Anonymous Coward

    Oh it gets better

    oh bless I was laughing all the way to work when I read this at Forbes.

    More like a tabby team. from the US army not a Tiger team

    See what happens when you don't have the correct wiring. they used wiring. And this lot think they are suitable to save the world... oh! you couldn't make this stuff up.

  48. John F***ing Stepp

    Has to be a color (colour) coding problem.

    "OK, tell me again, which black wire is hot; which black wire is ground; and which black wire is to the super-secret self destruct mechanism?"

    1. Blane Bramble

      Re: Has to be a color (colour) coding problem.

      That's classified information, citizen.

  49. This post has been deleted by its author

  50. Maharg

    “the civilian contractors at the sharp end of the project hired more than 30 independent experts to run 160 tests that chewed up 50,000 man-hours – without reaching a definitive conclusion about the cause of the problem, or how to prevent it.”

    I think we all know the issue here, too much evil, it’s just like the lightning coming out the eyes of the painting in that documentary I saw, Ghostbusters II

  51. Magnus_Pym

    feedback loop

    Maybe it's the spy-tech the NSA have forced manufacturers to build into the chips and stuff. All their kit is constantly trying to report back to itself creating an runaway feedback loop.

This topic is closed for new posts.

Other stories you might like