back to article Sandy takes out NYC colo facility

Post-tropical storm Sandy, as the event is now known, has taken out at least one piece of significant information infrastructure. Internap, which operates a global content delivery network and data centres to make it work, has reportedly emailed customers the following warning earlier today about one of its facilities in New …

COMMENTS

This topic is closed for new posts.
  1. Anonymous Coward
    Anonymous Coward

    wow...

    so, put your generators above ground, but your fuel reserves underground.

    They are probably under the servers, where a good explosion could wipe out everything, the data center ceo's "exit strategy".

    or.....

    We have just discovered where the BOFH works.

    In other news, the company I work for was very glad to hear that the datacenter our services are in is up without issue and none of our clients we are hosting there are affected.

    Although it did appear they lost power from the power companies, they were able to go to backup generators without a hiccup.

    1. Anonymous Coward
      Anonymous Coward

      Re: wow...

      Disaster = the wrong kind of rain.

      1. frank ly

        Re: wow...

        I'm sure 'lessons have been learned' so that this will never happen again.

        1. Anonymous Coward
          Anonymous Coward

          Re: wow...

          Don't diesel supplies tend to be underground? The whole pump thing from the trucks and all and the added safety of concrete?

          1. Anonymous Coward
            Anonymous Coward

            Re: wow...

            Indeed. And even quite cheap marine engines often have submersible electric pumps; I have a diagram of one somewhere in a textbook published in 1958, where an ordinary induction motor was housed in a deep bell. Fuel tanks tend to have ventilation pipes extending to a level at which you'd have to be sunk before water reached the fuel. It isn't exactly rocket science to ensure that your fuel system is flooding proof.

          2. David Cantrell
            Boffin

            Re: wow...

            Diesel doesn't go boom, but a big tank full of the stuff is bloody heavy, so you don't want it very high up.

          3. Fatman

            Re: Don't diesel supplies tend to be underground?

            The fuel tank underground, in and of itself isn't the prime issue here; it is the pumps that move the fuel from the underground tank up to the generator got submerged. Ideally, those pumps should have been located next to the generators, or at the same height as the generators, but not in a basement!!!

      2. amanfromMars 1 Silver badge

        Re: wow... Disaster = the wrong kind of rain. AC Posted Tuesday 30th October 2012 07:59 GMT

        For those in the know and in need to know circles, is that equation expandable to embrace and highlight ….. Disaster = the wrong kind of reign.

        And with the politically inept being so catastrophically adept in intellectually bankrupt administrations, are current systems destined for increasingly rapid terminal decline?

        And posed as a question for a ponder.

        1. Anonymous Coward
          Anonymous Coward

          Re: wow... Disaster = the wrong kind of rain. AC Posted Tuesday 30th October 2012 07:59 GMT

          remarkably terse for you... is it really you? ;-)

    2. Anonymous Coward
      Anonymous Coward

      Re: wow...

      Diesel is very, very unlikely to explode.

      Under the right conditions, it might burn fiercely.

    3. Terry Barnes
      WTF?

      Re: wow...

      So, in your world does fire tend to travel upwards or downwards? If it's upwards, you might want to consider why almost every building has fuel stores in the basement and water tanks at the top.

      Failure of the pumps is a problem - it might just not have been designed to work underwater - but the location of the fuel is absolutely correct. Generators below ground are a bad idea though as they're 'quite hard' to make work when submerged and the exhaust gases are heavier than air which can make undetected leaks a little dangerous for maintenance staff.

    4. Anonymous Coward
      Anonymous Coward

      Re: wow...

      Hey everybody! I've read an article about how a basement got flooded and I'm going to write comments about SHIPS because I'm a PRIZED CHUMP.

  2. Chris_Maresca

    The building in not a dedicated colo

    The tanks were likely underground as that was where building codes required them to be. Since fire is much more likely than a 100-year flood event, it's pretty sensible. Still, I fail to see why the pumps should fail...

    Anyway, I have servers in that colo - they went offline at exactly 8:33 PM PST, according to our monitoring. Here's a link to the location https://maps.google.com/maps?q=75+broad+st,+nyc&hl=en&sll=40.72586,-73.957644&sspn=0.050671,0.129175&gl=us&hnear=75+Broad+St,+New+York,+10004&t=m&z=17

    Also, it seems that trans-Atlantic cables are starting to go dark - https://twitter.com/skeevestevens/status/263137865578450944

    1. Chris_Maresca

      Re: The building in not a dedicated colo

      Everything came back up (as expected) about a 1/2 hour ago.

      I've used this same facility for the last 12 years across 4 different companies and the only other unplanned downtime was a failed switch for two hours about 6 years ago.

      This particular outage was a total of 7 hours during what is being described as a 'catastrophic' storm. Overall, I would say that 9 hours of unplanned downtime in 12 years is pretty good.

      1. Zaphod.Beeblebrox
        Meh

        Re: 9 hours of unplanned downtime in 12 years is pretty good

        That's only 4 9s uptime, so it depends on your SLA if that's pretty good or not...

        1. Chris_Maresca

          Re: 9 hours of unplanned downtime in 12 years is pretty good

          For a relatively cheap hosting, it's about as good as it gets. Even Amazon's vaunted redundant 'cloud' isn't as good.... Besides, a wealth of sites with vastly more resources went down as well, from Gizmodo to the Huffington Post, so I think we've done very well with our meager resources.

          The reality of infrastructure is that almost no one needs 'five nines' and even fewer people are willing or in a position to pay for it. It's a nice marketing term, but five-nines of the people who throw it around have zero need for it.

          Besides, in the modern world, you would want to achieve high-reliability through application architecture as well as good infrastructure. In our case, we could have failed over to a backup datacenter on the West coast, but the question was how long the NYC datacenter would be out, and if it was a good use of our time to migrate everything (it's a cold standby). In our case, the answer is no as we risked some data sync issues and our downtime was during the dead of night for all our customers (and they were all aware of the issues).

          Had it not come back up before end-of-day today, we might have considered differently. It's still touch & go, so we might migrate after all.

          1. P. Lee

            Re: 9 hours of unplanned downtime in 12 years is pretty good

            Indeed if you need really good uptime, you have another DC a long way away which you keep "hot".

            You don't do five nines with one data centre.

  3. Anonymous Coward
    Anonymous Coward

    A bit unlucky

    They allowed for some storage of fuel near the generators. Under normal circumstances 5 hours might have been enough to fix or workaround faulty pump but presumably they werent expecting them all to fail and the level to be flooded.

    Those who wanted more resilience in their server centre should consider locations that aren't ports.

    1. TheRealRoland
      Meh

      Re: A bit unlucky

      Possibly a similar situation as that hospital in New York - the backup generator was in the basement, which flooded. A backup to the backup was sitting high and dry, but the fuel pump that would supply the backup of the backup was located in that same basement that flooded...

    2. hplasm
      Coat

      Re: A bit unlucky

      A server centre without ports? At least it's secure...

  4. Dave Harris
    Alert

    IBM Triangle Park

    IBM's sites based at Triangle Park in North Carolina seem to be screwed somewhat (which is a bit of a bugger as I need some information from them)

  5. Anonymous Coward
    Anonymous Coward

    QWest and Time Warner

    QWest circuits in NYC down since 0249 GMT, while Time Warner circuits in NYC are down since 0212. Hibernia and Level3 still up. YMMV :-)

  6. Ian Johnston Silver badge
    WTF?

    Submarine cables affected by flooding

    Now there's a surprise. No, really.

    1. Anonymous Coward
      Anonymous Coward

      Re: Submarine cables affected by flooding

      I still remember when I found that submarine fibre cables along the west coast were affected by the autumn temp inversion. Nothing is really guaranteed to work all the time.

  7. Destroy All Monsters Silver badge
    Trollface

    In America!

    No Zombies involved?

    Not properly american, then.

  8. JaitcH
    Meh

    Why would anyone put assets in NYC when White Plains is so close by?

    White Plains, the East Coast international communications hub for decades, has an airport whose elevation is 439 feet (134 m) above sea level and is the terminus for many long haul fibre cables.

    It would take a very, very big surge to flood that place. And the BBC uses it to host it's services to North America from there, too.

    1. TheRealRoland

      Re: Why would anyone put assets in NYC when White Plains is so close by?

      Keep in mind, it's not only the surge - it may also be the torrential rain -- several inches in half an hour, and one of the entrances to a building built on (in?) a slight depression... That water wants to go somewhere.

  9. discontented
    Alert

    The real deal...

    The company I work for utilises Internap for our hosting needs. They have kept us up-to-date with regular updates as the night / morning has gone on. They originally sent an email stating they would have enough fuel for 65 hours but as the storm surge hit the Eastern coast the basement flooded and subsequently damaged the pumps that carry the fuel (basement level) to the generators (mezzanine).

    Egg on the face for Internap? Probably.

    The last update we had stated there was less than an hour until full lights out at the site.

  10. JohnMurray

    Video Of ConEd Station On FDR And 14th Street Exploding:

    http://www.youtube.com/watch?feature=player_embedded&v=ZAqYZ433TeQ

    http://www.zerohedge.com/news/2012-10-29/video-coned-station-fdr-and-14th-street-exploding

  11. Anonymous Coward
    Anonymous Coward

    Blow Day 2 (as it's called)

    (ahem)

    They (the COLO) might not have power because....

    1. Oyster Creek, NJ nuclear plant (They got diesels, are the PUMP underwater maybe, or close?)

    2. David Tillman said another Exelon reactor at the Limerick facility in Pennsylvania was reduced to 91 percent power after Sandy caused a problem with the condenser.

    3. Reuters: Just 6 more inches of water could submerge NJ nuclear plant’s pump to cool spent fuel pool — 25 hours to boiling without more coolant — New footage of flooding nearby

    4. More Nuclear Plants Affected: NY’s Indian Point reactor shut down — Storm causes condenser problem at Pennsylvania plant

    One of the units at Indian Point, a plant about 45 miles north of New York City, was shut down around 10:45 p.m. because of external electrical grid issues said Entergy Corp., which operates the plant.

    Are there more flooded pumps? Seems flooded pumps is a theme going here. Tsunami, Flooded Pump, Diesel Backup to burned up pump, Meltdown

  12. Anonymous Coward
    Anonymous Coward

    My anonymous, but very large global IT company has had to route our Europe > US transatlantic traffic via India and Australia, via the Pacific, to get to the US. Ouch.

  13. This post has been deleted by its author

  14. Fatman

    Management "strikes" again!!!

    This really cracks me up:

    Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.

    WHO in their fucking RIGHT MIND would allow fuel pumps to be located in a floodable basement????

    The only possible explanation is DAMAGEMENT!!!!!

    Call out the firing squad, now!

    1. Chris_Maresca

      Re: Management "strikes" again!!!

      Building codes - EOM

      1. TheRealRoland
        Unhappy

        Re: Management "strikes" again!!!

        Yup - i suspect these as well. You don't want the tank of fuel on top of a building; too heavy for one, and if it starts leaking, the whole building may be damaged.

        And that probably goes for pumps as well. If these pumps start leaking, it's better to have them leak in the basmement, rather than somewhere higher up.

        Guess you're damned either way: have a 'rare' weather event destroy stuff in basements and deal with the aftermath, or have several events with leaky pipes, pumps and containers to deal with, while hopefully during a rare event stuff remains working.

        Take that into a larger scale as well. The subways in NY are critical to everything going on - so why not spending a little more on prevention, rather than dealing with the aftermath?

        Again, it's probably something similar to 'why do this for a rare weather event', compared to spending money up-front.

        After a deadly flooding in 1953, the Dutch spent a whole lot of money on the Delta Works; preventing flood surges reaching the inlands. But, of course NL is only about twice the size of New Jersey, and the Dutch coastal line is again shorter as well -- so there's again that scale issue. Could there have been built higher (taller?) protection along the rivers, creeks, seafront all along the East Coast?

    2. Destroy All Monsters Silver badge
      WTF?

      Re: Management "strikes" again!!!

      Where else would you put the pumps? In the office level? I don't think so. Diesel dripping through the building tends to piss people off.

    3. LateNightLarry
      FAIL

      Re: Management "strikes" again!!!

      Not Damagement.... Manglement

      Either way, it's a F A I L...

  15. Anonymous Coward
    Anonymous Coward

    Fuel and gennys in basements

    In my experience working with several tier 1 telcos around the world is that generators and fuel are always at ground-level or in the basement. I guess some of the reasons are

    1/ Noise, easier to mask the noise of a generator when it's not placed on a roof but in a hole in the ground

    2/ Generators and the fuel are bloody heavy

    3/ Diesel doesn't explode but if it the tank leaks, it easier to control the flooded fuel when it's in a hole in the ground

    4/ If the fuel does somehow explode, better it explode in a hole in the ground that within the actual building

    Regarding the comments about 99.99r% uptime. No operator in the world can honestly, hand on heart, say they actually achieve this. It might say it in the SLA but that SLA also contains a lot of get out clauses, act of god etc. It's also a business risk / reward calculation, the cost of paying out liquidated damages versus the cost of actually achieving the supposed 99.999% means that it's cheaper to pay out damages now and again.

  16. MachDiamond Silver badge

    NY = poor location choice

    NYC is an expensive and rather stupid area to locate a data farm. Anywhere there is a good fat feed to the net and sufficient electrical power will do. The US's NSA set up their data center to store "Big Brother" info in Utah. It makes sense; cheap land, low probability of earthquakes, no flooding issues and relatively zombie free. The USA's SS is, unfortunately, not stupid. I would sleep better if they were.

    1. Barry Mahon

      Re: NY = poor location choice

      +1, probably the most intelligent comment on this thread. OTOH the original decisions were probably based on the (lack of?) availability of (secure?) circuits to a further away facility. OK for the NSA, they can spend and spend.

      OOTOH the military/nsa/cia etc., have poor form on this sort of thing too.

      Maybe the best comment is $=refer upwards=careful=justify it=unlikely?=reduce budget.

  17. Anonymous Coward
    Anonymous Coward

    There's always a reason they want to host there...

    The providers in question all have large state of the art facilities well away from these sorts of dangers. It's their clients who want NYC location, not them. Some of them have good reason (financial types like the low latency), some have practical reason (small companies taking colo like locations near their offices as people will need to visit the datacenter every so often and it's convenient that way) and some are idiots who think it's the only way to get good connectivity.

    These sites are not ideal for hosting but they're popular for logical reasons and an absence of good alternatives.

    Also worth noting, Peer1 are still online there as they've somehow managed to manually carry enough diesel from street level to the 17th floor (no elevators of course) to keep their generators running this far...credit to them for the effort...

This topic is closed for new posts.