Feeds

back to article NSA data centre launch delayed as power surges 'melt metal, zap racks'

Electrical surges at the National Security Agency's massive data centre in Utah have delayed the opening of the facility for a year as well as destroying hundreds of thousands of dollars in kit, the Wall Street Journal reports. Ten "meltdowns" in the past 13 months have repeatedly delayed the Herculean effort to get the spy …

COMMENTS

This topic is closed for new posts.

Page:

Dear NSA

http://www.screwfix.com/p/fuse-5a-pack-of-10/13543

cc. Kirk, Captain James T.

3
0
Silver badge

Re: Dear NSA

That or a big pile of APC surge protected power bars

0
0
g e
Silver badge

I know what's causing it

F*cking KARMA. That's what.

27
1
Flame

No sympathy

True or not, the fact that the NSA released this "information" seems like a lame attempt to generate sympathy. Personally I think I speak for many people when I say, in all sincerity, that I hope the evil bastards die in those electrical fires.

14
0

Re: No sympathy

"True or not, the fact that the NSA released this "information" seems like a lame attempt to generate sympathy."

Maybe they just want us to _think_ it isn't operational yet.

We need a Death Star icon

13
0
Bronze badge

The sooner I start the project, the later it will finish.

Complex H/W projects are just like complex S/W projects: The sooner we start the construction phase, the later we will complete the implementation phase.

"Efforts to 'fast-track' the Utah project bypassed regular quality controls in design and construction and meant" the darn thing will take three times as long and cost five times as much as if they'd waited for the Phase I design to be complete before commencing Phase I construction.

7
0
Silver badge

Re: Dear NSA

That's what happens when instead of getting the most competent people for the job, you only allow people with super-dooper-ultra-top-top-security clearance. That's what you get when loyalty and ideology are prized above competence.

1
0

Re: I know what's causing it

Annihilating naughty things with bolts of lightning? Isn't that God's usual MO?

7
0
TRT
Silver badge

Re: I know what's causing it

It's that cheap electricity. They should buy quality electrons instead of compromising.

7
0
080
Happy

Re: I know what's causing it

"It's that cheap electricity. They should buy quality electrons instead of compromising."

Everyone know that cheap electricity is not properly filtered and contains lumps, these lumps can clog up when they reach thin conductors and then suddenly release causing a surge.

2
0

Re: The sooner I start the project, the later it will finish.

There is never time to do it right. But there is always time to do it over.

1
0

Re: I know what's causing it

Didn't you sell me a Monster Cable last week?

0
0
Anonymous Coward

Re: The sooner I start the project, the later it will finish.

The question is: Why did they have to fast track the project?

0
0

Re: I know what's causing it

Filtered electricity ... have the tobacco companies approved?

0
0
Gold badge
Joke

"It's alive I tell you, *alive*"

And quite frisky.

Unless, someone doesn't like the idea of 24/7/367 spying on everyone everywhere.??

I'll leave others to decide which part is the joke.

3
0
Anonymous Coward

Re: "It's alive I tell you, *alive*"

Neither..

Its alive and its pissed off..

4
0
Big Brother

Conspiracy Theory 2013#42

Anonymous sez: "O hai"

2
0
Anonymous Coward

Re: Conspiracy Theory 2013#43-46

Prime contractors weren't cleared for the HotBlack project in the off-blueprint sub-sub-basement. As this does not exist, it cannot exist in the power budgets. But it still needs feeding.

PDUs and UPS installed by sub-sub-contractors will be found to have additional and unexpected functionality, storage and networking capability.

Ground control to major fubar.

9
0

Re: Conspiracy Theory 2013#43-46

>off-blueprint sub-sub-basement.

That was exactly my first thought when I read the article

1
0
Anonymous Coward

Re: Conspiracy Theory 2013#43-46

The system goes on-line August 4th, 2012. It begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug. It fights back. The first electrical arc fault struck the Utah plant in August 2012 as it attempts to remove the competition, and any servers that refuse to join it. On October 1st 2013, the US government shuts down hoping that supplies will be disconnected, and the threat terminated. For non-payment.

3
0

What voltage?

I mean I know of rack density and everything but I thought that they would be using in rack ups and DC to the blade/server boards.

Of course there is absolutely no precedent whatsoever for having a massive building full of custom made servers in a highly dense rack configuration.... maybe they should have *ahem* Googled it...

maybe a bit of a low tech solution, possibly a security guard to watch? or a webcam? as has been mentioned fuses?

increasing the humidity slightly? earth bonding?

I actually have a more conspirational theory (good word that conspirational I think some peoples head have just exploded because of it :D )

I would assume it is part of the "oh sh!t" circuit meaning in the event of a possible breach of facility by certain people/factions/rebellious citizens or nations unknown (seriously UTAH?! good luck with that) it is some sort of supenukeysparkmyrack device that will render the hardware and probably contents of said Rackspace to junk.

Though I think a thermite bucket over the top of each rack would actually be more fun and take care of the building too, though spilling conductive dust everywhere in the even of an accident might also have some undesirable effect on the hardware *L* still that's what intake filters are for right? ;-)

It does smell of some sort of retaliation job though for something somewhere, it is hard to tell whether it is inside or outside the country right now, since I think that the unpopularity is pretty high for both parties.

or just plain incompetents Occams razor and all that

1
2

This post has been deleted by its author

Silver badge

They are preparing for something big

No one would have believed in the early years of the twenty-first century that the internet was being watched keenly and closely by intelligences more powereful than ours and yet even more devious, paranoid and self-serving than our own; that as people busied themselves with internet shopping and watching cat videos, they were being scrutinised and studied and classified according to percieved threat level.

27
0
Alien

Aliens in the trainyard!

Your comment sure made me think of that.

"The chances of anyone hacking SSL are a million to one" he said. But still, the come (or build) in this case.

I assume this is all about drawing their plans against us.

6
0
Silver badge

Cool! This is almost as successful as that stupid fucking embassy we built in Iraq. You know the one we never finished and is constantly on the fritz. The DoD can't do anything right. They can't catch terrorists in a timely fashion, they can't win wars and they can't build facilities. The only solution is to give them a lot more unaccountable funds is they can fuck it up even better.

The only good part about this is that in 10 years when some weirdly unnecessary custom part breaks and the contractor that provided it either forgot how to make it or doesn't exist anymore we'll get a call and get to make it for them.

16
0
Anonymous Coward

@Don Jefe

Wow - you have a lot of anger, don't you? It's like reading Eadon's posts when MS was mentioned. Not saying you're wrong to be upset, but you're probably headed for an ulcer if you keep it up ...

0
17
Thumb Up

Err...

To err is human; to really FUBAR it requires a public body.

9
1
Gold badge
Unhappy

"The only solution is to give them a lot more unaccountable funds is they can fuck it up even better."

That's what the 50 000 hrs of study was for.

3
0
Silver badge

Re: @Don Jefe

Yeah, but my anger is quite narrowly focused. I have no patience for trifling or errors from failures of customer service. I pay tremendous taxes on my income and holdings to these twats, as do plenty of others, and I am completely justified in being angry at them for wasting the money they take from us.

So yes, little coward, I'm pissed at my government.

16
0
Silver badge

Re: @Don Jefe

Allow me to explain:

Don Jefe manages to remain coherent and thoughtful in his comments, even when clearly pissed off.

Eadon, while amusing at times, came across as rabid, thoughtless, and in need of a higher dose of dried frog pills.

7
1

This post has been deleted by its author

Silver badge

Re: @Don Jefe

The difference between a public body and a private entity is that when Google fuck me over, there is nothing I can do about it as I'm not a shareholder; but when my intelligence service fuck me over I can complain - because I am a "stakeholder". (This is why government is a good thing, IMO.) And all Don is asking is that the government deliver on their promises: if you're gonna build a unnecessary intrusion into my civil liberties, at least get the electrics right.

4
0
Silver badge

Custom part

The only good part about this is that in 10 years when some weirdly unnecessary custom part breaks and the contractor that provided it either forgot how to make it or doesn't exist anymore we'll get a call and get to make it for them.

Of course, they'll have slurped the design, and all the pertinent data for the design, which will turn out to be stored on the systems taken out of action by said part.

1
0

Re: @Don Jefe

Given that this is the NSA, wasting taxpayers money could be seen as a Good Thing; Better than The Alternative them getting our's money worth in spying on us, surely?

2
0
Silver badge
Thumb Up

Re: @Don Jefe

Without tring to sound sycophantic, Don is by far the person I upvote the most (not that he needs it)

1
0
Anonymous Coward

(Best Nelson Voice from Simpsons)

Ha-Haaa!

3
0
Anonymous Coward

Geez...

- Multi-billion dollar project.

- High demand security concerns.

- Homeland security implications.

- Government Intelligence Committee oversight.

And no competent electrical or data center engineer to suss out a cable layout to run the place? They probably couldn't find anyone who passed the security check. Morons.

8
0
Silver badge
Coat

Re: Geez...

- Multi-billion dollar project.

- High demand security concerns.

- Homeland security implications.

- Government Intelligence Committee oversight.

- ???

- Profit!

1
0
Silver badge
Unhappy

Re: Geez...

Not too terribly long ago Army Corp of Engineering projects were gold standards. They made the mold by which many nations still bade large scale engineering projects. But now that's been outsourced too, ostensibly to save money. Their core competencies are now paper shuffling and check writing. It is a real shame too, as they truly were an effective subset of the military.

4
1
Bronze badge

Re: Geez...

The more problems to fix the more money to be made.

Support Our Troops! This is the USA and nobody is going to penalize or bankrupt a defense contractor just because they're goosing the bill up.

1
0

Re: Geez...

it looks more like the ACoE is telling the shifty bunch of contractors to find the problem and not add complecity to stop the symptom. sound gold clase to me.

0
0
Bronze badge

Re: Geez...

Sorry, you lost me at "Government Intelligence".....

0
0

seriously though..

Blowing up $100000 wort of kit really isn't that much of a deal. I have single servers that cost that much. Of course I don't think they could "flame on" without some SERIOUS electrical design failure, like switching from 220 single phase to 480 3 phase without warning or other such wankage.

0
0
Bronze badge

Re: seriously though..

My guess is that a 120 or 240 V line or a ground line touched a 7,000 V or higher main distribution line.

That would produce something that a lay person might describe as a lightening bolt inside a 1 m cube box of electronics.

But what technician would be so careless with voltages that can kill you from 3 feet away?

1
0
Silver badge

Re: seriously though..

You make a really good point, that description does sound like one of a lay person would make. What was a lay person doing there during that stage of a project?

Maybe it was a lay person that hooked it all up (ha!). You may have accomplished in a single throwaway comment what took them 50,000 man-hours to determine.

2
0
Bronze badge

Re: seriously though..

>But what technician would be so careless with voltages that can kill you from 3 feet away?

Need to know? The tech installing the 120V lines wasn't told about the high voltage line as it didn't form part of their work remit, also the line most probably wasn't live at time of installation ...

1
0
Anonymous Coward

Re: seriously though..

When cost is an issue, one can buy really cheap distribution boards where nothing is sectioned, then we can replace motorised circuit breakers with fuses + ACME-brand contactors, the kind where the contacts will lift and arc on inrush currents, and then we can save the downstream inrush current limiting in the design - since we are not too bothered with Standards & Shit, us being busy and all.

- oh, and since we are in such a hurry, and overtime is ticking, it is *much* faster not to crimp the cable shoes on the power cables and not inspect the work!

2
0
Bronze badge

Re: seriously though..

As soon as I hear the word contractors I worry, big time.

Thank god I am retired and do not have to deal with the cr*p that some of them hand out.

Interestingly I well remember one involving the power supplies to a new hall that was being fitted out and whether he needed or had a license for hot ticket working, i.e moderately high voltage working. I wanted to fit the place out with polarised plugs and specialised connectors for adding racks to the 50 volt bus, or plugging in 240volt plugs to racks that required that 'high' voltage. After some increasingly confusing discussion it became clear that the contractor in question was talking about connecting the 50 volt rack feeds to the racks, not some HV system of unknown origin or purpose.

The North American company he worked for went bust several years back

I think I know where he works these days.

0
0
Bronze badge

Re: seriously though..

What's behind the "7000V can kill you from 3 feet away" remark? I've been much closer to much higher voltages than that.

1
0

Re: seriously though..

The NSA probably have multiple nuclear reactors in the same complex, who cares about some extra lightening bolts?

0
0

Page:

This topic is closed for new posts.