Feeds

back to article Inside Facebook's engineering labs: Hardware heaven, HP hell – PICTURES

Facebook's hardware development lab is either a paradise, a business opportunity, or a hell, depending on your viewpoint. If you're a hardware nerd who loves fiddling with data centre gear, ripping out extraneous fluff, and generally cutting the cost of your infrastructure, then the lab is a wonderful place where your dreams are …

COMMENTS

This topic is closed for new posts.
Bronze badge

Send reg reporters on an engineering refresher

I think you'll find that's a mill, not a lathe.

10
0
Bronze badge
Stop

Re: Send reg reporters on an engineering refresher

Googling metal lathe appears to shoe El Reg is correct :(

I havent downvoted as I also thought it was a milling machine before spotting the additional handles.

0
1
FAIL

Re: Send reg reporters on an engineering refresher

No headstock, no tailstock, not a lathe.

4
0
Anonymous Coward

Re: Send reg reporters on an engineering refresher

> Googling metal lathe appears to shoe El Reg is correct :(

Short of it being some sort of combo unit, I am pretty sure that's a milling machine. As a general rule if the motor is on top it isn't a lathe.

Lathes spin the piece against cutter whereas milling machines have a spinning cutter. If that is a lathe then it's an alarmingly badly designed one...

5
0

Re: Send reg reporters on an engineering refresher

Definitely a Bridgeport Vertical Milling Machine - no way its a lathe - Google Image it...

And, yes, using an anti-static bag (which is conductive, albeit only slightly) as the base testing a PCB is nuts.

8
0
Anonymous Coward

Re: Send reg reporters on an engineering refresher

Definitely Bridgeport, I've got one sitting in my house complete with power table feed.

Like to get rid of it though... good steel, scraps for around 600. Just don't use it enough to justify the 150 a year it costs in taxes on the space it takes up.

0
1
Bronze badge

Re: Send reg reporters on an engineering refresher

Don't bin it...advertise it on Craigslist, free for the taking.

Somebody will come and pick it up. Those things are still much in demand. Beats the hell out of the cheap Harbor Freight version.

0
0
Silver badge
Paris Hilton

Re: Send reg reporters on an engineering refresher

to justify the 150 a year it costs in taxes

You mean even capital infrastructure is being taxed these days (in addition to the universal tax one has to pay to entropy?)

"Oh Zimbabwe, soon we there..."

0
0
Silver badge

Re: Send reg reporters on an engineering refresher

"And, yes, using an anti-static bag (which is conductive, albeit only slightly) as the base testing a PCB is nuts."

The mettalised mylar type in the photo are only conductive on the inside. The outside can build up some quite startling charges.

0
0
Silver badge

Re: Send reg reporters on an engineering refresher

"I think you'll find that's a mill, not a lathe."

Who cares? This is a site for computer people, not metal workers.

Bloody pedants.

0
0

50's Lathe

That would be a milling machine. At last my CAD/CAE education has found a use, 22 years later!

6
0
Bronze badge

Re: 50's Lathe

Hey, mistakes are easily made. I honestly thought the 3D printer was a coffee machine before I read the description.

Interesting article, though. FB has gone up a bit in my estimation, allowing a bit of hardware creativity within its ranks.

1
0
Bronze badge

Additive and subtractive manufacturing

One machine to lay the bits down, one slice at a time..

One machine to carve bits out, one chunk at a time.

Or holes, slots keyways, etc..

And I'll bet somewhere behind the camera they've got a laser cutter to make enclosure sides...

0
0
Coat

!!STOP THE PRESS!!

"The 3D printer, for instance, is linked to Facebook's network so that engineers can order up print jobs from their desk."

A printing device has been attached to a network.. Whatever next, a device that you hold to move your cursor around the screen?

10
0
Bronze badge
Facepalm

Re: !!STOP THE PRESS!!

Yeah, I think you missed the point. Unlike the rest us poor schmoes who are lucky if we're allowed to print a sheet of A4 without some HR/stationary nazi coming down on us for not thinking, these FB lads can print what they like, when they like and in 3D if they so choose and no one will tell them off.

8
0
Silver badge

Re: !!STOP THE PRESS!!

3D is the new colour laser?

6
0
WTF?

A server development board being tested (page1 pic)

On (what looks like) an anti-static bag. Really?

7
0

Re: A server development board being tested (page1 pic)

^ditto

testing a board on a surface designed to conduct electricity....

5
0
Silver badge

Re: A server development board being tested (page1 pic)

I "like" this!

0
0

It's a mill.

Oh. Too late.

3
0
Anonymous Coward

Cool!

Good article. Nice to see hardware hackers still have a role - challenging the major vendors by doing things better, smarter, cheaper, faster...

2
0
Bronze badge

Re: Cool!

There is actually considerably less innovation shown here than it seems. Most vendors have drive sleds with multiple drives deep, although I think most vendors avoid attaching that many SSDs because in normal land we use SSD for bandwidth and that many SSDs on a SAS loop would kill this PDQ. It's currently not possible to have sufficient CPUs in a box that size to allow enough SAS loops to saturate those disks either. Given that Facebook are after massive capacity and lower power they would be considerably better off manufacturing new SSD disks a few inches thick with more chips in them. Fewer SAS connections would lower the power requirements further while a 5" thick SSD would be large enough to incorporate its own cooling solution. It would also reduce their costs for engineers replacing them because they would fail less often and there would be fewer to swap out on failure. Given their desire for extremely low access speed and high capacity I'm also amazed they don't write custom firmware (maybe they do?) to utilise the spare capacity in SSDs which is usually reserved for replacing dead blocks. In enterprise drives there is double the stated capacity usually so this would be a much more worthwhile project for them than cocking about with 3D printers and quadcopters. Of course, I don't have billions of dollars to my name so clearly they are smarter than I am :)

5
1
Anonymous Coward

Re: Cool!

The point of the exercise is to knock things up quickly and cheaply from commodity components. Extensively customised componentry kinda defeats the objective.

2
0
Silver badge

Re: Cool!

The point of the exercise is to knock things up quickly and cheaply from commodity components.

Yeah, but they did it by defining their own commodity components in way of the open compute project. All that stuff is custom, but its now "facebook custom".

If facebook said "We're going to order $100m worth of 5" SSDs each year for the next 10 years at least", there would be companies falling over themselves in Taiwan to assemble them for them, and anyone else using OCP spec kit.

0
0

Question on server types

Article mentioned 3 of 5 (Database, Photo, and Hadoop) server types. Other two?

1
0

Re: Question on server types

Front-end Web & News Feed. Bit more info here (PDF) http://opencompute.org/summit/ [note - in this slide preso Haystack == Photo]

1
0
Silver badge
Facepalm

Meanwhile, outside hobbyville....

".....When these designs are published, the Asian companies are usually the ones selling the designs – at the cost of the profits of HP, Dell, Lenovo, and so on....." Yeah, right up until the customer stops to think about support, and what it means when their cheapo datacenter falls over and they can't get support unless they talk Chinese and can afford to wait around whilst half-a-dozen component manufacturers try and work out how their products fit together. At that point, the reality is businesses go buy hp, Dell, Lenovo, NetApp, EMC, etc., etc., who supply properly tested, integrated and supported products.

1
5

Re: Meanwhile, outside hobbyville....

Well..... sort of.

Seeing as it's built out of commodity parts, you can either have a bin full of spares to cope with hardware failure, or since you built the thing, you'll have a really good idea how to fix it when it breaks. That having been said:

YES, It's certainly not for small businesses or for those without a clue (or for medium to large enterprises that demand 24/7 support from their vendors and also demand tried and tested hardware with similar support from the vendors.

However, if you have the talent in-house and they are willing to support it at 2 am when it falls over and needs fixing ASAP, then that's a different story.

2
0

Re: Meanwhile, outside hobbyville....

The point is as is built from commodity parts you simply replace what is broken or bin the thing.

0
1

Re: Meanwhile, outside hobbyville....

Isn't Lenovo a chinese company Matt?

3
1
Bronze badge

Re: Meanwhile, outside hobbyville....

>However, if you have the talent in-house and they are willing to support it at 2 am (or 24x7x52) when it falls over and needs fixing ASAP, then that's a different story.

Or alternatively, if you need to have talent sitting around at 2 am to fix stuff ASAP, perhaps it is cost effective to use their skills to make it easier to fix stuff ASAP...

0
0
Bronze badge

Re: Meanwhile, outside hobbyville....

Or you have so much kit that if device X breaks, the system recovers and keeps going via automated processes. Manual intervention should only be required to fix the broken device and restore that device to service - the fix may involve replacing a few parts or replacing the unit with the current model.

Maybe a new definition is needed for "cloud services" - a service consisting of clusters of data centres where losing any physical service up to and including a data centre does not result in a loss of service. Not all cloud service providers will meet this, but they should get somewhere close.

0
0
Silver badge
Facepalm

Re: ToddR Re: Meanwhile, outside hobbyville....

"Isn't Lenovo a chinese company Matt?" Yes, but they actually have a 24x7 support service that actually knows about more than just their own kit, documentation in English, labs that do proper integration testing (going on hearsay here), and provide COTS stacks.

I have done plenty of Linux DIY projects in enterprise settings, the problem usually starts when you hit an issue that affects all your systems, which means just dragging out another box out of the cupboard is not going to solve your problem. Usually it's because you updated something to get a new feature or because a vendor of one part of your solution released new drivers to fix a possible problem (such as a security hole). So you go to the maker of that product and - if you're lucky - they say stick a fix on, only when you test it something else on another vendor's bit of the stack breaks. So you go talk to that vendor (if you can), and they say they have never seen the problem, remove the other vendor's update or go without support. But you can't, because the business wants that new feature or security fix, so you have to hack a workaround that may break something different elsewhere, and so on and so on. That is the reality of kludging together untested stacks. Sure, if you can find and employ a team of gurus to build and maintain your stack you may survive, and if you're doing nothing more than posting webpages, and you can afford the time to play around with plywood trays, you could prosper, otherwise COTS, tried-and-tested stacks with proper support make a lot more sense to the business.

2
0
Bronze badge

Re: ToddR Meanwhile, outside hobbyville....

>the problem usually starts when you hit an issue that affects all your systems, which means just dragging out another box out of the cupboard is not going to solve your problem.

We've already seen this problem with cloud providers eg. Azure last autumn...

0
0
Anonymous Coward

pah

it's unlikely facebook will ever come to anything. a few servers in a warehouse run by amateurs. no one even uses the site.

3
0
Anonymous Coward

Now this... is engineering...

This is cool stuff! :-)

0
0
Bronze badge

Re: Now this... is engineering...

Not half as cool as going to the next stage and actually building your own motherboards etc.

One of things about the IT industry in the late 70's and early 80's was the number of companies who were building their own hardware and hence also required their own software engineering team. Hence to me the absence of scope's, ICE and other test systems from the Facebook lab's is quite telling.

0
0

An event horizon for economists?

Talking about external suppliers and 'In-House' expertise is an interesting exercise in economics: we know it makes sense for small and medium-sized companies to use external expertise for almost every 'non-core' skill and activity.This is less true for larger companies - but big companies still contract-out everything from catering to networks and desktop support.

Are they wrong? Or is there a size of company that's so big that they are a nation unto themselves and their internal IT functions are bigger than IBM? Or, at least, bigger than any hardware supplier and service provider that would take on an S&P500 company's IT?

It's staggering that an IT software and service provider - and that's what Facebook is - is so large that it 'in-sourcing' the hardware supply budget can generate internal economies of scale that exceed the efficiency of the world's biggest hardware suppliers. That's supposed to be an economic absurdity - one company, one hardware customer, is a bigger market than the entire customer base of the world's biggest hardware supplier, or their second and third-largest rival?

Is there no advantage to specialisation at all? Or some kind of economic 'event horizon', beyond which space and time and economics curve around and disappear when a gravitation amassment of money exceeds some critical threshold?

Basic economics reads the FaceBook 'in-house' hardware supplier as a signal that the IT hardware industry has a *lot* of room for consolidation, and efficiency gains, and for trimming excessive profit margins. Except that competition should've done that already, if economists are right. And we've already got a heavily consolidated industry with dominant suppliers and razor-thin margins.

Could it be that consolidation and shrinking margins have left us with unresponsive monopolists and no money for investment in cheaper products and more efficient production?

Meanwhile, management: is a company that size actually manageable? As in: controllable in terms of internal direction and supervision?

It goes without saying that we've long since passed the event horizon for external supervision by governments and the rule of law to prevent abusive monopolies, regulatory capture, and tax evasion.

3
1
Silver badge
Facepalm

Re: An event horizon for economists?

That's supposed to be an economic absurdity - one company, one hardware customer, is a bigger market than the entire customer base of the world's biggest hardware supplier, or their second and third-largest rival?

It's called "specializing for a special task". It is possible if commodity stuff can be thrown together to build a specialized tool as materials and know-how become commponplace. It's the same reason why there are not only 5 companies building houses in the world.

It goes without saying that we've long since passed the event horizon for external supervision by governments and the rule of law to prevent abusive monopolies, regulatory capture, and tax evasion.

I don't know what world you are living in, but we are frankly into Nazi territory and moving deeper regarding:

1) Arbitrary antitrust shit whenever some (generally leftish) cunt in government feels like it "Uh... I think people are collunding ... how horrible. Let's reduce the price forcefully for the peons!!" Sure a formula for Great Success. As opposed to letting the monopoly die on its arse by enabling some competition.

2) Regulatory capture being ENABLED by government at full throttle. Checked whether you company is "green enough" lately? Yep, you are being forced to dump the lightbulbs for no good reason.

3) "Tax evasion" being chased down like Himmler's mystical projects depended on it. Which is why you hear about it so much. These taxfeeders are voting themselves pay increases and are HUNGRY.

0
0
Bronze badge

Re: An event horizon for economists?

Nah. Just too many 'safe-pair-of-hands' middle managers. Who will only buy ripoff Cisco and HP kit because they like crawling handcuffed into sacks to take a kicking. Buying commodity stuff without outrageously overpriced warranties or useless support contracts is a no-no. Safe is expensive.

0
1
Silver badge

Re: An event horizon for economists?

I suppose if you're spending $3bn in opex, sometimes it can be worth speculatively spending $50m in capex to try and reduce that to $2bn, but delivering 4 times the capacity.

Plus facebook imagines itself a disruptive company. You don't pay IBM $50m when you can put some geniuses in a shed with some credit cards - much more disruptive.

1
0

I actually like what FB are doing, and should be commended for allowing their staff to be creative and find solutions to problems without having to have degree's in who knows what and somehow get through to someone at the top of the top of the chain. Encouraging staff to FIND and create a better system for them is great for them and at the same time a real boost for staff when their idea becomes used and is successful. Well done FB, a lot of respect earnt by me for this .

0
0
This topic is closed for new posts.