back to article New York tech firms form 'bucket brigade' to fuel flagging servers

Three technology firms have joined forces to avoid any data center downtime in the aftereffects of Hurricane Sandy amid the continuing power outages crippling Lower Manhattan. Employees of Peer 1 Hosting, blog host Squarespace, and Fog Creek Software have formed a 'bucket brigade', lugging diesel up to the backup generators …

COMMENTS

This topic is closed for new posts.

Page:

Fukishama

So they've put the generators away from harm, but not the pumps that supply the fuel. A bit like Fukishama.

1
0
Boffin

Re: Fukishama

There's this slight problem with putting pumps up high, called gravity. It's the same reason you can't suck water up 10m with a straw or why a barometer will have empty space up top despite the vacuum. Simply put, there's only so high atmospheric pressure will push the diesel despite how good the pump is sucking, a problem not faced by a pump that's pushing. So you have to have the pump as low as the tanks, or else they won't work beyond the third floor, if you're lucky.

5
4
Silver badge

Re: Fukishama

How do oil wells work then?

You can pressurize the tanks and have the pumps wherever you like providing the pressure to push the fuel up. You can also have hydraulic pumps where the motor and hydraulic pump part are high up and the drive pump is in the sump.

The reason not so suck diesel (and other fuels) with a vacuum pump is that you would evaporate off all the volatiles, leaving you with tar in the tank and a motor full up fumes.,

1
5

Re: Fukishama

Offshore oil wells work off the natural pressure of the well at first (from the compression that in fact made the oil) so no pumping/sucking at all. Later in the well's life when the pressure is insufficient to naturally come to the well head and rig they actively pump water down into the well to increase pressure and push the "last few bits" (usually actually quite a lot) up.

1
0
Anonymous Coward

Re: Fukishama

Obviously you've not installed much less used any kind of pump. In the land of Fukushima, we have oil pumps for kerosene on, you guessed it, the floor it's consumed on. Water pumps for wells are installed, not at the bottom of the well, but you've guessed it again, at top of the well or even higher. Gosh, that is farking amazing! Pumps! Whoda thunk...

1
6
Silver badge

Re: Fukishama

Water pumps for wells are installed, not at the bottom of the well, but you've guessed it again, at top of the well or even higher. Gosh, that is farking amazing! Pumps! Whoda thunk...

You just made that up. I operate my own water system, as does everybody else where I live. So yes, I have installed pumps - many times. A shallow well has the pump at the top. A deep well has the pump at the bottom. As mentioned above, air pressure is limited and quite low so you would not get any water out of a deep well if you had the pump at the top.

7
1
Vic
Silver badge

Re: Fukishama

> So you have to have the pump as low as the tanks

Not *quite* as low as the tanks; you can suck the diesel a short distance.

Whether or not that would be enough to make a difference, I really wouldn't know...

Vic.

0
0
Vic
Silver badge

Re: Fukishama

> How do oil wells work then?

That depends on the well.

Some are under sufficient pressure that the oil will flow to the surface without pumping

If that is not sufficient, a pump is sent down the shaft, driven from the surface.

> You can pressurize the tanks

No-one sane pressurises the tanks.

> providing the pressure to push the fuel up

Now do some calculations to see how much pressure you'd need to move diesel up the requisite number of floors. Fancy that in a diesel tank?

> The reason not so suck diesel (and other fuels) with a vacuum pump is that you would evaporate

This is not true. The OP was correct - the maximum you can get out of a lift pump is determined by ambient air pressure, and that will only move diesel around 12m if everything is perfect. And nothing is ever perfect...

Vic.

3
0
Gold badge

Re: Fukishama

Little bit of bollocks getting talked here. Actually enormous amounts. From memory water can only be sucked up 7m, at normal atmospheric pressure, so you could only have the pump 7m above the tank, that's 2 floors. I don't know the specific gravity of diesel. It floats on water, so it's going to go a few meters higher. Vic says 12m, and his post makes sense, so I'll go with him.

Also, using suction lift on a backup generator is risky. Your whole system basically relies on the footer valve, which is usually a spring loaded check valve / non-return valve at the bottom of the pump inlet pipe. If that spring doesn't close properly, then diesel will leak back into the tank, to be replaced by air, and the pump will lose prime, and won't work. And then your back-up system will fail. I don't sell pumps on suction lift, if I can persuade the customer not to, because of the hassle they always seem to cause. Also, isn't diesel always a bit dirty? I do water, not other stuff, so this isn't my area. But vital valve with spring closure and liquid with bits in = bad combination and leaky valve.

You DON'T pressurise your diesel tank. What if it sprung a leak? Erk! Anyway how would you fill it? How would you have an overflow pipe? Not a good idea.

Any well deeper than say 5m will have a submersible pump lowered down it on a chain, and that will pump the water up to the top. The controls and inverter (if variable speed) will live at the top of the well, but the pump and pump motor at the bottom.

Normal pumps in a basement won't be water-proof, it's very difficult to waterproof motors that are air-cooled.

Stick submersible pumps in your diesel tank? Well I don't deal with fuel, but you're sticking a big old electrical thing in a tank of fire-risk. I know diesel doesn't burn that easily, but even so. Also submersible pumps dump their heat into the surrounding liquid. No probs when the tank is full, or being topped up from a pump, but as the tank gets empty, and non tanker truck arrives, you're warming up a decreasing supply of diesel in a large container with an electrical item. I don't know the rules for doing this, it's not my area. I suspect there are lots of them though.

2
0
Silver badge

Re: you're warming up a decreasing supply of diesel

in a large container with an electrical item.

I smell a Mythbusters episode!

0
0
Gold badge

Re: Fukishama

Can I just add by the way:

Tank = vessel holding liquid under pressure

Cistern = vessel holding liquid at atmospheric pressure (i.e. with vent / no lid).

Everyone calls both a tank anyway. Even in the industry. Because confusion is fun...

1
1
Happy

Squarespace

sent out a cicular last night saying things were bad.

Access tonight is a little slow but it's still there at 2125 GMT.

0
0
bep

They should use 44 gallon drums

They'd be lighter!

0
0
Pint

55 Gallon Drums of Diesel

They're lugging 55 gallon drums of Diesel fuel up 17 floors by hand (presumably without elevators, since the building doesn't have any electricity)?!? Good &deity!!! Those things will be about 385 pounds (175 Kg)! I salute their efforts.

Dave

P. S. I hope they have a bunch of cold, frosty beverages lined up after work like that.

4
0
Paris Hilton

Won't anyone think of the HSE?

I'm wondering what health & safety thinks of that, hand-carrying part-emptied drums upstairs through semi-darkness, outside working hours with people that have worked far beyond their normal shifts.

But then again, I think if it helps some blogs about cats staying online that's worth a few lives.

1
1

Re: Won't anyone think of the HSE?

I suspect they dont care and are concerned with Real issues.

0
0
Silver badge
WTF?

wow

I guess some people like their employer enough to do hard manual labor hauling a very dangerous, flammable carcinogenic substance. As for me I ain't that guy. I personally would be taking care of my family and property instead.

1
3
Silver badge

Re: wow

Yes diesel is harder to ignite but it does contain more energy per volume that gasoline I believe. Regardless even though its a common substance in our environment if you look at a hazmat sheet on either and you would realize they are some of the more dangerous chemicals most people ever deal with.

1
0
Pint

Re: wow

Still, I take my hat off to those guys who are trying to provide a service to their customers despite the odds. They all deserve a BIG cold frosty

9
0
Orv

Harder to ignite...

...is a bit of an understatement. Once, while using diesel fuel to light a fire, I dropped a match into a puddle of it. It put out the match.

1
0
Silver badge
WTF?

Re: wow

Well, as long as you don't breath it in and keep it off your skin, you should be good. It's not classified as a carcinogen either (though diesel burn products are):

PDF

"In a multi-site, case-control study, there was evidence for an increased risk of prostate cancer and squamous cell carcinoma of the lung [36] but this effect could not be attributed to any particular chemical. The International Agency for Research on Cancer (IARC) have evaluated diesel fuels as being “not classifiable as to their carcinogenicity to humans (Group 3)” [9]: there is “inadequate evidence” to classify diesel as a human carcinogen and “limited evidence” for the carcinogenicity of diesel to experimental animals.

What is amazing is that generators are on the upper floors. I seriously doubt that is a good idea. What if a fuel line breaks?

1
0
Silver badge

Re: Harder to ignite...

Diesel can be burned like wax. Dip something porous and somewhat flammable in it (like a rag, paper, whatever), and burn that. In fact I've seen candles that use diesel instead of wax, with the wick on a floating thing atop a pool of the stuff. Cheaper than lamp oil, I guess.

0
0
Happy

Re: wow

It's not too uncommon for high-rise buildings to have service floors periodically. That allows water pumps and tanks to be installed. Otherwise, an incredible pressure would be needed to pump water to the top of some of those buildings, pressures high enough that it would burst common pipes. And, for that matter, turning on a faucet at a lower level would produce a stream of water so intense that it could cut things (seriously hard things, like granite and steel). Thus, to avoid problems like this, it's common to put holding tanks and helper pumps on service floors of high-rise buildings.

Additionally, the water storage tanks aid in fire fighting by providing a reservoir of water for sprinklers and fire hoses. This source of water even works without electrical power, which may be compromised in the event of a disaster.

Furthermore, the water in the storage tanks can be used as a dynamic mass damper to reduce the tendency for the building to resonate when excited by earthquakes, or even by the wind.

The service floors also make a convenient place to put elevator motors for the banked elevators which service the floors below the service floor.

The service floors may also be convenient places to put electric distribution/switching panels, and/or transformers. So, it would be somewhat logical to put backup generators in the same space.

This concludes this architecture lesson; we now return you to the normal commenting.

Dave

P.S. Mine's the one with a copy of "The Tower and the Bridge" by Donald P. Billington in the pocket. I attended a lecture by Professor Billington once; it was quite interesting.

10
0
Anonymous Coward

Re: wow

A big salute to these folks. Just hope that their company recognizes and remembers their efforts.

1
0
Silver badge

Re: Harder to ignite...

Thanks for putting us straight. I can't believe the earlier comments making statements about diesel that are completely wrong. In fact I wonder why somebody who obviously doesn't work with the stuff would venture a comment. Embarrassing, really.

4
0
Headmaster

Re: Harder to ignite...

I have designed furnace burners that run on diesel - and there are two ways to do it.

One is to spray a heavy fog of diesel onto a big pile of lit paper and rag inside a furnace and that lights up the spray, until the furnace gets hot enough for the fuel to ignite by temperature alone.

The second way is to preheat the diesel prior to spraying it, just like the kerosene based blow torches - and let me tell you something - it has to be REALLY REALLY REALLY fucking hot to autoignite on contact with air as a spray.

(guessing a bit here - I have the tables stashed away)

I think it's natural flash point is about 250*C or 280*C.

Something like that.

0
0
Silver badge

Re: Harder to ignite...

You CAN do the same thing with gasoline. In both cases it is the fumes which ignite, not the liquid. It is admittedly much more difficult to do with a bucket of gas since it has a lower temperature for volatility, but it is possible. And yes, this would fall under the category of "Don't intentionally try this at home."

0
0
Silver badge
Trollface

Re: Harder to ignite...

TBH I wouldn't haul water bottles forget drums of it for an employer. Went to college to avoid having to do that.

0
0
Silver badge

Fog Creek s/w

Isn't that Joel Spolsky of "Joel on Software"?

1
0
Anonymous Coward

Re: Fog Creek s/w

That's the one. Plus CEO of StackOverflow (et al).

1
0
Facepalm

Suck versus Blow

To everyone who yammers on aghast about the pumps being in the basement, and how it's a horrible design: there's a very simple reason why they're there. It's called basic physics. If you are trying to get fluid from a low level to a high level (and please God don't try to suggest the diesel tanks should be anywhere but the basement) you have two ways to do it: suck or blow. If you want to suck it up (i.e., have the pumps up high) you can only raise it up until the head pressure equals the vapor pressure of the fluid. For example, if you're trying to suck mercury up a tube, you can only raise it 760mm (assuming you're at sea level). After that, you simply have a vacuum over the top of the liquid (actually mercury vapor, but that's pretty close to a good vacuum). You simply can't suck it higher than that. Water, (and diesel), being less dense than mercury, can get sucked up higher, but you can only go so high before again, you only get vapor, not fluid.

The other approach is to blow. Put the pump inlet down at the lower level, and then you can send it as high as the outlet pressure of the pump can push it. Given that pumps can easily hit hundreds of PSI (and thousands in the case of hydraulics), you can push a liquid far, far further than you can pull it.

So for the love of all things holy, please quit saying how horrible a design it was to put the pumps in the basement. It's the only place they can really do their job.

However, feel absolutely free to question why the pumps weren't waterproof (or submerged in the diesel tanks), along with watertight electrical feeds to them. Now THAT is a horrible design.

3
2

Re: Suck versus Blow

So why not pump air into the tank in the basement to push the diesel up to the roof where the generators and pumps could be. Surely this is a pushing approach but using pumped air rather than atmospheric pressure?

If you're worried about air in the fuel lines then have another temporary tank on the roof to allow any air bubbles to vent before being used

1
3
Silver badge
Pint

Re: Suck versus Blow

"why the pumps weren't waterproof (or submerged in the diesel tanks), along with watertight electrical feeds to them",

I suppose the problem is no electricity right now.

Bye the way, sucking pumps that suck higher have to pipes one bigger and one smaller and what you get is the difference between the size of the pipes. The bigger pipe brings it up and the smaller lets the rest back down. You can suck higher with this system but hardly this high.

0
2
Unhappy

Re: Suck versus Blow

So, let me get this right, you want to pump an oxidizer (air) into a tank full of flammable fuel? Note that compressing air heats it up. For that matter, that's exactly how a Diesel engine works, by compressing air to about 10 atmospheres of pressure, where it becomes hot enough to ignite the Diesel fuel.

Dave

5
1

Re: Suck versus Blow

""why the pumps weren't waterproof (or submerged in the diesel tanks), along with watertight electrical feeds to them",

I suppose the problem is no electricity right now."

As long as you had enough fuel to get the generators running then they could power the pumps too.

1
0
Anonymous Coward

Re: Suck versus Blow

Physics and energy levels are ALL wrong.

0
0
Anonymous Coward

Your company doesn't run on diesel...

it runs on people. I can see Fog Creek knows that, and the other two. Why is it that nearly every other company fails to 'know' that? In the past I've waded across a flooded bayou to show up and help keep a data center going. (Forgot about the snakes) But can I even imagine doing that now? You can replace me - I can replace my job - I *can't* replace *me*.

4
0
Silver badge
Childcatcher

Don't say "your company runs on people"

The way business execs and the economy are these days, you'll probably find some brightly lit datacenter with curiously few employees around, and the generators are being run on a biodiesel fuel that the CEO calls "oilent green"

7
0
Flame

Helo

fly it in to to roof.

chopper it in to the roof

when the winds die down

0
1

Re: Helo

with a small civilian helicopter (and you would not be able to land a heavy military one) you would be lucky to get 4-5 drums in (using previous commentards 175kg figure) not to mention the danger involved and the sheer price of doing it that way.

1
0
Pint

Use the cloud

Why do they not have redundant Cloud? Fast, reliable, and not affected by disaster, war, political discontent. Though in this case, it will still be hard to not have your cloud become part of the hurricane. Maybe we should think of having a more inland alternate backup Wall Street. The military has multiple redundant headquarters. How about the Poconos? Any site suggestions?

0
2
Trollface

Re: Use the cloud

The cloud ain't going to carry buckets of fuel up the stairs, no matter how hard you pray it can.

3
0
Silver badge

Re: Use the cloud

The idea is that you have a backup set of servers with exactly the same data etc on them somewhere else that wasn't affected by the hurricane. Maybe the West coast, or Australia or something like that. Then it doesn't matter that your New York servers are down due to lack of power.

0
0
Boffin

Re: Use the cloud

Some web sites have done that.

StackExchange.com (including StackOverflow.com) are running out of their backup DC in Oregon, some of Fog Creek's applications are now in AWS.

Of course this isn't something you can just magically do, throw some servers into another DC and it will work, you need both application changes and administrative processes (and monitoring) that will allow the fail over. (Eg. StackExchange recently – in a rather timely manner – tested their fail over, found some issues and fixed them.) Whether for a given web site it is worth the costs for the level of risks is a business choice.

0
0
FAIL

Re: Use the cloud

jonathanb, you've clearly never costed disaster recovery options. Fully mirrored systems don't cost QUITE as much as the original system, but you still have hardware, software, networking & maintenance costs.Totally out of the question for smaller organisations.

0
0

So sad

At least 68 people are dead and likely more. Lives and property destroyed forever. The damage pictures are surreal.

God Speed people...

1
1
Anonymous Coward

Buckets?!?

I cannot believe they are not using fuel-canisters/jerry-cans/containers/whatever-you-call-them-things.

0
0

Curiously, only a few people above demonstrated knowledge of basic physics.

You can draw water only about 20 feet up into a pump. In theory a little under 10 meters, but practical pumps can't draw a hard vacuum and easily cavitate. Because diesel is a little less dense than water, you can draw 15% further, perhaps 4 fathoms, but that's a minor effect. We'll call that two office floors, not three.

Most water wells are deeper than this and must use submersible pumps to push up rather than draw from the top. That's why the well casing is so large: so that a large-ish pump can be lowered down (plus to trap an occasional curious child).

The old fashioned hand pumps have a rod extending to a lift disk near the bottom of the well.

Pressurizing the basement tanks would allow pumping from the top. Although at that point you can go slightly further and just use the air compressor. But now you have a highly pressurized tank of fuel. It would have to be immensely strong for safety, and any tiny leak is a disaster. Far worse, if you use regular air the partial pressure of oxygen becomes a major problem. Even cold, the fuel will spontaneously decompose, ignite and explode.

The pictures make it clear that they are transferring the oil from the 55 gallon barrel to 5 gallon buckets. A pair of those is a reasonably effective load for a man to carry. Even in excellent shape, you won't get in many 17 floor trips before you call it a day.

4
0
Anonymous Coward

"Curiously, only a few people above demonstrated knowledge of basic physics."

Not particularly curious, I suspect the only people that know basic physics are 40+.

Welcome to the dawn of modern education, now how to do bold text in Excel

4
0
Silver badge

Re: only people that know basic physics are 40+.

and even then, if it isn't your day job, you likely have forgotten much of it. I forgot about the atmospheric pressure angle on the pump even though I do know how a barometer works and that mercury was originally chosen specifically because of its high specific gravity.

And I thank all those who did remember for their explanations.

As this story proves yet again, its awfully difficult to make something foolproof, because fools are so damned ingenious. Even the non-sentient ones.

1
0

Page:

This topic is closed for new posts.

Forums