Feeds

back to article Sandy takes out NYC colo facility

Post-tropical storm Sandy, as the event is now known, has taken out at least one piece of significant information infrastructure. Internap, which operates a global content delivery network and data centres to make it work, has reportedly emailed customers the following warning earlier today about one of its facilities in New …

COMMENTS

This topic is closed for new posts.
Anonymous Coward

wow...

so, put your generators above ground, but your fuel reserves underground.

They are probably under the servers, where a good explosion could wipe out everything, the data center ceo's "exit strategy".

or.....

We have just discovered where the BOFH works.

In other news, the company I work for was very glad to hear that the datacenter our services are in is up without issue and none of our clients we are hosting there are affected.

Although it did appear they lost power from the power companies, they were able to go to backup generators without a hiccup.

0
3
Anonymous Coward

Re: wow...

Disaster = the wrong kind of rain.

1
2
Silver badge

Re: wow...

I'm sure 'lessons have been learned' so that this will never happen again.

3
0
Anonymous Coward

Re: wow...

Don't diesel supplies tend to be underground? The whole pump thing from the trucks and all and the added safety of concrete?

7
0
Anonymous Coward

Re: wow...

Diesel is very, very unlikely to explode.

Under the right conditions, it might burn fiercely.

9
0
Anonymous Coward

Re: wow...

Indeed. And even quite cheap marine engines often have submersible electric pumps; I have a diagram of one somewhere in a textbook published in 1958, where an ordinary induction motor was housed in a deep bell. Fuel tanks tend to have ventilation pipes extending to a level at which you'd have to be sunk before water reached the fuel. It isn't exactly rocket science to ensure that your fuel system is flooding proof.

3
0
Bronze badge
WTF?

Re: wow...

So, in your world does fire tend to travel upwards or downwards? If it's upwards, you might want to consider why almost every building has fuel stores in the basement and water tanks at the top.

Failure of the pumps is a problem - it might just not have been designed to work underwater - but the location of the fuel is absolutely correct. Generators below ground are a bad idea though as they're 'quite hard' to make work when submerged and the exhaust gases are heavier than air which can make undetected leaks a little dangerous for maintenance staff.

3
0
Silver badge

Re: wow... Disaster = the wrong kind of rain. AC Posted Tuesday 30th October 2012 07:59 GMT

For those in the know and in need to know circles, is that equation expandable to embrace and highlight ….. Disaster = the wrong kind of reign.

And with the politically inept being so catastrophically adept in intellectually bankrupt administrations, are current systems destined for increasingly rapid terminal decline?

And posed as a question for a ponder.

4
3
Anonymous Coward

Re: wow... Disaster = the wrong kind of rain. AC Posted Tuesday 30th October 2012 07:59 GMT

remarkably terse for you... is it really you? ;-)

1
0
Boffin

Re: wow...

Diesel doesn't go boom, but a big tank full of the stuff is bloody heavy, so you don't want it very high up.

2
0
Anonymous Coward

Re: wow...

Hey everybody! I've read an article about how a basement got flooded and I'm going to write comments about SHIPS because I'm a PRIZED CHUMP.

1
9
Bronze badge

Re: Don't diesel supplies tend to be underground?

The fuel tank underground, in and of itself isn't the prime issue here; it is the pumps that move the fuel from the underground tank up to the generator got submerged. Ideally, those pumps should have been located next to the generators, or at the same height as the generators, but not in a basement!!!

1
0

The building in not a dedicated colo

The tanks were likely underground as that was where building codes required them to be. Since fire is much more likely than a 100-year flood event, it's pretty sensible. Still, I fail to see why the pumps should fail...

Anyway, I have servers in that colo - they went offline at exactly 8:33 PM PST, according to our monitoring. Here's a link to the location https://maps.google.com/maps?q=75+broad+st,+nyc&hl=en&sll=40.72586,-73.957644&sspn=0.050671,0.129175&gl=us&hnear=75+Broad+St,+New+York,+10004&t=m&z=17

Also, it seems that trans-Atlantic cables are starting to go dark - https://twitter.com/skeevestevens/status/263137865578450944

5
0

Re: The building in not a dedicated colo

Everything came back up (as expected) about a 1/2 hour ago.

I've used this same facility for the last 12 years across 4 different companies and the only other unplanned downtime was a failed switch for two hours about 6 years ago.

This particular outage was a total of 7 hours during what is being described as a 'catastrophic' storm. Overall, I would say that 9 hours of unplanned downtime in 12 years is pretty good.

4
0
Meh

Re: 9 hours of unplanned downtime in 12 years is pretty good

That's only 4 9s uptime, so it depends on your SLA if that's pretty good or not...

2
0

Re: 9 hours of unplanned downtime in 12 years is pretty good

For a relatively cheap hosting, it's about as good as it gets. Even Amazon's vaunted redundant 'cloud' isn't as good.... Besides, a wealth of sites with vastly more resources went down as well, from Gizmodo to the Huffington Post, so I think we've done very well with our meager resources.

The reality of infrastructure is that almost no one needs 'five nines' and even fewer people are willing or in a position to pay for it. It's a nice marketing term, but five-nines of the people who throw it around have zero need for it.

Besides, in the modern world, you would want to achieve high-reliability through application architecture as well as good infrastructure. In our case, we could have failed over to a backup datacenter on the West coast, but the question was how long the NYC datacenter would be out, and if it was a good use of our time to migrate everything (it's a cold standby). In our case, the answer is no as we risked some data sync issues and our downtime was during the dead of night for all our customers (and they were all aware of the issues).

Had it not come back up before end-of-day today, we might have considered differently. It's still touch & go, so we might migrate after all.

0
0
Silver badge

Re: 9 hours of unplanned downtime in 12 years is pretty good

Indeed if you need really good uptime, you have another DC a long way away which you keep "hot".

You don't do five nines with one data centre.

0
0
Anonymous Coward

A bit unlucky

They allowed for some storage of fuel near the generators. Under normal circumstances 5 hours might have been enough to fix or workaround faulty pump but presumably they werent expecting them all to fail and the level to be flooded.

Those who wanted more resilience in their server centre should consider locations that aren't ports.

1
0
Meh

Re: A bit unlucky

Possibly a similar situation as that hospital in New York - the backup generator was in the basement, which flooded. A backup to the backup was sitting high and dry, but the fuel pump that would supply the backup of the backup was located in that same basement that flooded...

1
0
Silver badge
Coat

Re: A bit unlucky

A server centre without ports? At least it's secure...

2
0
Alert

IBM Triangle Park

IBM's sites based at Triangle Park in North Carolina seem to be screwed somewhat (which is a bit of a bugger as I need some information from them)

0
0
Anonymous Coward

QWest and Time Warner

QWest circuits in NYC down since 0249 GMT, while Time Warner circuits in NYC are down since 0212. Hibernia and Level3 still up. YMMV :-)

0
0
WTF?

Submarine cables affected by flooding

Now there's a surprise. No, really.

4
1
Anonymous Coward

Re: Submarine cables affected by flooding

I still remember when I found that submarine fibre cables along the west coast were affected by the autumn temp inversion. Nothing is really guaranteed to work all the time.

0
0
Silver badge
Trollface

In America!

No Zombies involved?

Not properly american, then.

0
0
Silver badge
Meh

Why would anyone put assets in NYC when White Plains is so close by?

White Plains, the East Coast international communications hub for decades, has an airport whose elevation is 439 feet (134 m) above sea level and is the terminus for many long haul fibre cables.

It would take a very, very big surge to flood that place. And the BBC uses it to host it's services to North America from there, too.

3
1

Re: Why would anyone put assets in NYC when White Plains is so close by?

Keep in mind, it's not only the surge - it may also be the torrential rain -- several inches in half an hour, and one of the entrances to a building built on (in?) a slight depression... That water wants to go somewhere.

0
0
Alert

The real deal...

The company I work for utilises Internap for our hosting needs. They have kept us up-to-date with regular updates as the night / morning has gone on. They originally sent an email stating they would have enough fuel for 65 hours but as the storm surge hit the Eastern coast the basement flooded and subsequently damaged the pumps that carry the fuel (basement level) to the generators (mezzanine).

Egg on the face for Internap? Probably.

The last update we had stated there was less than an hour until full lights out at the site.

1
0

Video Of ConEd Station On FDR And 14th Street Exploding:

http://www.youtube.com/watch?feature=player_embedded&v=ZAqYZ433TeQ

http://www.zerohedge.com/news/2012-10-29/video-coned-station-fdr-and-14th-street-exploding

0
0
Anonymous Coward

Blow Day 2 (as it's called)

(ahem)

They (the COLO) might not have power because....

1. Oyster Creek, NJ nuclear plant (They got diesels, are the PUMP underwater maybe, or close?)

2. David Tillman said another Exelon reactor at the Limerick facility in Pennsylvania was reduced to 91 percent power after Sandy caused a problem with the condenser.

3. Reuters: Just 6 more inches of water could submerge NJ nuclear plant’s pump to cool spent fuel pool — 25 hours to boiling without more coolant — New footage of flooding nearby

4. More Nuclear Plants Affected: NY’s Indian Point reactor shut down — Storm causes condenser problem at Pennsylvania plant

One of the units at Indian Point, a plant about 45 miles north of New York City, was shut down around 10:45 p.m. because of external electrical grid issues said Entergy Corp., which operates the plant.

Are there more flooded pumps? Seems flooded pumps is a theme going here. Tsunami, Flooded Pump, Diesel Backup to burned up pump, Meltdown

0
1
Anonymous Coward

My anonymous, but very large global IT company has had to route our Europe > US transatlantic traffic via India and Australia, via the Pacific, to get to the US. Ouch.

1
0

This post has been deleted by its author

Bronze badge

Management "strikes" again!!!

This really cracks me up:

Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.

WHO in their fucking RIGHT MIND would allow fuel pumps to be located in a floodable basement????

The only possible explanation is DAMAGEMENT!!!!!

Call out the firing squad, now!

2
0

Re: Management "strikes" again!!!

Building codes - EOM

0
0
Silver badge
WTF?

Re: Management "strikes" again!!!

Where else would you put the pumps? In the office level? I don't think so. Diesel dripping through the building tends to piss people off.

0
0
FAIL

Re: Management "strikes" again!!!

Not Damagement.... Manglement

Either way, it's a F A I L...

0
0
Unhappy

Re: Management "strikes" again!!!

Yup - i suspect these as well. You don't want the tank of fuel on top of a building; too heavy for one, and if it starts leaking, the whole building may be damaged.

And that probably goes for pumps as well. If these pumps start leaking, it's better to have them leak in the basmement, rather than somewhere higher up.

Guess you're damned either way: have a 'rare' weather event destroy stuff in basements and deal with the aftermath, or have several events with leaky pipes, pumps and containers to deal with, while hopefully during a rare event stuff remains working.

Take that into a larger scale as well. The subways in NY are critical to everything going on - so why not spending a little more on prevention, rather than dealing with the aftermath?

Again, it's probably something similar to 'why do this for a rare weather event', compared to spending money up-front.

After a deadly flooding in 1953, the Dutch spent a whole lot of money on the Delta Works; preventing flood surges reaching the inlands. But, of course NL is only about twice the size of New Jersey, and the Dutch coastal line is again shorter as well -- so there's again that scale issue. Could there have been built higher (taller?) protection along the rivers, creeks, seafront all along the East Coast?

0
0
Anonymous Coward

Fuel and gennys in basements

In my experience working with several tier 1 telcos around the world is that generators and fuel are always at ground-level or in the basement. I guess some of the reasons are

1/ Noise, easier to mask the noise of a generator when it's not placed on a roof but in a hole in the ground

2/ Generators and the fuel are bloody heavy

3/ Diesel doesn't explode but if it the tank leaks, it easier to control the flooded fuel when it's in a hole in the ground

4/ If the fuel does somehow explode, better it explode in a hole in the ground that within the actual building

Regarding the comments about 99.99r% uptime. No operator in the world can honestly, hand on heart, say they actually achieve this. It might say it in the SLA but that SLA also contains a lot of get out clauses, act of god etc. It's also a business risk / reward calculation, the cost of paying out liquidated damages versus the cost of actually achieving the supposed 99.999% means that it's cheaper to pay out damages now and again.

0
0
Bronze badge

NY = poor location choice

NYC is an expensive and rather stupid area to locate a data farm. Anywhere there is a good fat feed to the net and sufficient electrical power will do. The US's NSA set up their data center to store "Big Brother" info in Utah. It makes sense; cheap land, low probability of earthquakes, no flooding issues and relatively zombie free. The USA's SS is, unfortunately, not stupid. I would sleep better if they were.

0
0

Re: NY = poor location choice

+1, probably the most intelligent comment on this thread. OTOH the original decisions were probably based on the (lack of?) availability of (secure?) circuits to a further away facility. OK for the NSA, they can spend and spend.

OOTOH the military/nsa/cia etc., have poor form on this sort of thing too.

Maybe the best comment is $=refer upwards=careful=justify it=unlikely?=reduce budget.

0
0
Anonymous Coward

There's always a reason they want to host there...

The providers in question all have large state of the art facilities well away from these sorts of dangers. It's their clients who want NYC location, not them. Some of them have good reason (financial types like the low latency), some have practical reason (small companies taking colo like locations near their offices as people will need to visit the datacenter every so often and it's convenient that way) and some are idiots who think it's the only way to get good connectivity.

These sites are not ideal for hosting but they're popular for logical reasons and an absence of good alternatives.

Also worth noting, Peer1 are still online there as they've somehow managed to manually carry enough diesel from street level to the 17th floor (no elevators of course) to keep their generators running this far...credit to them for the effort...

0
0
This topic is closed for new posts.