back to article Facebook's request to the flash industry: 'Make the worst flash possible'

Flash-memory designers may currently be focused solely on improving speed and endurance, but Facebook's director of capacity engineering and analysis wants something completely different from them. "The Facebook ask of the industry is make the worst flash possible," Facebook's Jason Taylor told his keynote audience at the …

COMMENTS

This topic is closed for new posts.
  1. ObSolutions, Inc

    I'd buy that

    ...gods only know how many GB cold storage I have sitting around in my closet, on spindles that are old, slow and outdated - but I don't dare throw away, because I might need that data someday.

    It would make the missus happy too.

    1. ObSolutions, Inc
      Happy

      Re: I'd buy that

      (something caused that to get cut short)

      It would make the missus happy too, seeing as how much closet space would be recovered.

      1. Arbee
        Joke

        Re: I'd buy that

        > (something caused that to get cut short)

        It was probably the full stop.

        1. amanfromearth

          Re: It was probably the full stop

          Or a period. That usually causes those kinds of activity to stop for a few days.

    2. Khaptain Silver badge
      Coat

      Re: I'd buy that

      "but I don't dare throw away, because I might need that data someday."

      No need to worry, 1980's porn has evolved, today we have moving pictures........

      PS : I have upgraded, replaced all of my drives, work and home, with SSDs. ( Well except for the the USB archive/backup attached to the NAS). I don't even remember the sound of the click, click, scratch, screech as data is being trawled from the anals of the highly defragged IDE 30 Mb "whopper" of a drive......

      1. Lusty Silver badge
        Paris Hilton

        Re: I'd buy that

        "anals of the highly defragged IDE 30 Mb "whopper" of a drive......"

        Hmmm anals and whopper eh? Maybe use annals and big next time...you got Paris all excited.

        1. Khaptain Silver badge

          Re: I'd buy that

          "Hmmm anals and whopper eh? Maybe use annals and big next time...you got Paris all excited."

          Lol, The spell checker was feeling horny....

      2. Destroy All Monsters Silver badge
        Windows

        Lolita on my drive, oh my!

        > 1980's porn has evolved

        At least the women back then looked like women.

        1. Anonymous Coward
          Anonymous Coward

          Let's grumble about the 80s...

          "At least the women back then looked like women."

          No, that was the 70s. The fake, over-made-up look was already becoming dominant by the 80s, and ironically the overstyled hair and makeup styles of 80s porn look *way* more horribly dated nowadays.

        2. Amorous Cowherder
          WTF?

          Re: Lolita on my drive, oh my!

          "At least the women back then looked like women."

          No they all looked like failed Cybil Sheppards, too much make-up, big hair and half the vaseline budget went on covering the camera lens for soft-focus shots!

          Now they don't even bother with all that guff, drug up a couple of 15 year-old runaways on Sunset Strip, get them to "act" in a hastily rented house, set up a Nikon D3/Canon 5D3 on a stready-cam shoulder mount and start shooting HD porn! (*)

          (*) Note that I'm not an expert on shooting low-budget adult fare, regardless of the investigation I've done when I'm bored!

    3. Peter Gathercole Silver badge

      Re: I'd buy that @ObSolutions

      Tell me, how do you attach a ST-506 or ST-412 drive to a modern machine? You can't even plug in the ISA controller card into any machine built in the last 10 years or so.

      I mean, even EIDE and SCSI are disappearing rapidly.

    4. Anonymous Coward
      Anonymous Coward

      Re: I'd buy that

      Many years ago, I went through all my old hard drives, user-created CDs and DVDs, SD cards, and USB flash drives, and basically just dumped their contents onto my (at the time) extremely large 1TB hard drive and then erased them and donated them to charity or threw them out, as appropriate.

      After the most rudimentary de-duplication by hand (i.e., only copy my user folder from old hard drives, and try not to end up with 18 copies of my entire MP3 collection), the entire unorganized collection ended up being less than 200GB. The process required an adapter to mount a bare hard drive via USB ($20) and maybe 2 afternoons at most. I made a big directory called "backups" with a bunch of subdirectories called e.g. "White SanDisk 2008" and then just copied away.

      The benefits are obviously less clutter, plus now I only have one main drive with all my data to secure/back up, plus I can search for documents from 15+ years ago in seconds, plus I'm not worried that if somebody breaks into my home they might find a loose unencrypted drive with valuable personal information. Highly recommended.

  2. Blain Hamon
    Coat

    So what he's asking for is...

    Write Once, Read Few. These WORF drives could be the Next Generation of backups, reducing the constant trekking from one format to the next, and could be quite good for Enterprise.

    Right, I'll go now.

    1. Dan 37
      Go

      Re: So what he's asking for is...

      Indeed they could, we have a fair bit of rarely used Data we just have to Klingon to.

      1. Jim 59

        Re: So what he's asking for is...

        "Write-once, read-never is probably the spec for a lot of this

        It's called /dev/null.

        Snarky comments aside though, intriguing article.

    2. Darth_RayDar
      Coffee/keyboard

      Re: So what he's asking for is...

      I'll bet that WORF joke's been in cold storage a while. Maybe in the transporter buffer? Beam ... up Scotty!

    3. Jerren
      Coat

      Re: So what he's asking for is...

      Make it so...

      Yeah I'm leaving now as well...

    4. Tom 13
      Devil

      Re: So what he's asking for is...

      Good idea. But I think Paramount has a trademark on the name, so you'll need a new one.

  3. Version 1.0 Silver badge

    WORN

    Write Once, Read Never - I have a stack of these tapes at home - I believe that the complete source code to RSX11-M 4.3 is on one of them.

    1. Isn't it obvious?

      Re: WORN

      I thought that was mounted at /dev/null.

      I put the JBoss logs there, and anything else that wants to write megabytes of garbage I'm never going to look at.

    2. Mephistro Silver badge
      Thumb Up

      Re: WORN

      My thoughts precisely. Read WORN's datasheet here.

      1. Eddy Ito Silver badge

        Re: WORN

        Great datasheet, pity it's going to be blown away by modern virtualization technology.

  4. Anonymous Coward
    Coat

    Isn't it already ?

    For a minute I thought this was a dig at Adobe ...

    1. Tom 35 Silver badge

      Re: Isn't it already ?

      Yes, it's the return of Punch the Monkey!

  5. LaeMing Silver badge
    Thumb Up

    I'd buy that for a dollar!

    I have an immediate use for such tech. Has to have long store times and high density, but not at all fussy about speed or re-writes above a few hundred cycles.

    1. Malcolm 2
      Black Helicopters

      Re: I'd buy that for a dollar!

      So will the NSA ...

      1. yourpastcatchingupwithyou
        Holmes

        Re: I'd buy that for a dollar!

        So will the NSA ...

        Erm isn't that what the article was about?

        Facebook does with consent what the NSA does without? One and the same............

  6. malle-herbert Silver badge

    How long do you want to keep that data ?

    Because, just like magnetic media slowly loses it's magnetic polarisation due to the earths magnetic field,

    flash-memory loses it's electrical charge over time.

    Okay... it may take 10 to 20 years but still... the data may need to be read and re-written every 5 years or so...

  7. Lyle Dietz
    Pint

    The BOFH is precognisant

    I remember him talking about the magic of WORN drives may years ago

  8. Graham Dawson

    So he wants write-only memory?

  9. Graham Marsden
    Holmes

    "Write-once, read-never"

    But think of all the thousands of pictures of teh kitties!!!

  10. Zola

    "The majority of that data will probably be written once and read never – really, it's sad."

    You can say that about pretty much all user generated content on the web - Facebook just happens to be storing by far the most of it this write-only shite.

    I'd describe the whole of Facebook itself as "very sad", but then I never saw the point of it when it first appeared, and still don't today.

    1. Eddy Ito Silver badge
      Thumb Up

      Re: "The majority of that data will probably be written once and read never – really, it's sad."

      You heard 'em, it's the shite pile of stuff you don't want to toss but also never want to deal with.

    2. Lusty Silver badge
      Pint

      Re: "The majority of that data will probably be written once and read never – really, it's sad."

      "I'd describe the whole of Facebook itself as "very sad", but then I never saw the point of it when it first appeared, and still don't today."

      Well aren't you the cool one for avoiding it. Here, let me get you started:

      - No longer a need to stand and pose while 17 cameras are used so everyone gets a picture of an event

      - No need to hope your mates will share pictures of events

      - One update and everyone knows the plan, no need to call around, text around etc.

      - If plans change everyone sees immediately and nobody is left out of the loop (except you, you're not on FB)

      - I don't need to travel 10,000 miles a week or spend hundreds on phone bills to keep in touch with friends

      - My photo album is enormous, and I know who is in the pics because they are all tagged, and I can contact them immediately when reminiscing over the photos

      - After Uni everyone went separate ways and contact is easily lost. Facebook means I always have an up to date phone directory and email address

      - FB groups have replaced Yahoo groups for communities and clubs. It's amazing the people you meet online when you share interests

      - Sharing an interesting link with friends

      I'm sure there are more, that was just off the top of my head a few things to help you see the point. Of course you're one of the cool kids so will now explain how you can do all of that but using email and ICQ chat in twice the time with half the friends...

      Also worth noting that a friend of mine back home often gets left out of things because someone needs to remember to call or text him - everyone else just sees the broadcast and comes to the pub.

      1. Getriebe
        Windows

        Re: "The majority of that data will probably be written once and read never – really, it's sad."

        @Lusty - I'm old, very old - possibly enough to be your granddad but me and my mates use FB in that way - and its a damn sight easier to get every one to arrive at a damp field somewhere in Europe (either to climb a lump of rock or look at each other's cars) than any other method.

        Sorry if I have now made it uncool for you as well.

    3. Timmay
      Stop

      @ Zola

      Fashionable to hate Facebook, isn't it.

      If you've tried it, genuinely don't like it, then fine, I respect that. But, there are about a billion of us (I guess you'd call us idiots) who do have a use for it.

      If, however, you're just an old miser who likes to moan about everything, then fuck off back to the good old days of the 19th century, where women and the poor knew their place, England ruled the world, and the only content which was "user generated" was quite literally shit.

      (Twitter, on the other hand, even I don't get...)

      1. Zola

        @ Timmay: Re: @ Zola

        If you've tried it, genuinely don't like it, then fine

        Yes I tried it, recently in fact (earlier this year) as I recalled a lot of friends that were raving about it way back in 2010, so I looked up their entries and guess what - not one of them has posted a single update since about 2011/2012.

        I looked up about two dozen people and there's just tumble weed rolling through their pages.

        I'm sure Facebook has its uses, and appeals to certain groups, but like the guy from Facebook said - "The majority of that data will probably be written once and read never" - which chimes with my personal experience, nobody (I know) is using it any more. They've all moved on, Facebook for them was just a fad.

        (Twitter, on the other hand, even I don't get...)

        I actually find Twitter quite useful, following people (not necessarily friends) that have interesting opinions or commentary. Although anyone posting more than half a dozen tweets a day will usually get the chop in no short order.

  11. chris lively

    Just to be clear

    The Director of Capacity Engineering at the largest social site on the planet, a company which not just begs to have people constantly upload stuff but actually does everything it can to tie everything about you in a single location has just said that most of it is crap that no one will *ever* see or care about.

    The takeaway from this for FB users should be "No one cares. Really, they don't."

    1. RaidOne

      Re: Just to be clear

      Upvoted. It makes sense, who the heck has the time to look at all the pictures that all "friends" are uploading like crazy? What matters for Facebook is that the user feels he/she is important and keeps uploading pictures and so on, seeing ads in the same time.

      1. Rune Moberg

        Re: Just to be clear

        Some content is temporary in nature.

        E.g. two weeks ago I posted a picture of a burger with a note that says "and this joint have wi-fi". I knew that a friend of mine would be travelling the same way a week later and my picture gave him a feel for what kind of food to expect and what other services were available. And sure enough, a week later he made a stop at that location. I had also tested a shortcut which I told him about and he subsequently followed.

        So a week ago my silly burger picture had value, but now it has become largely irrelevant and a year from now possibly just garbage. For all I know, the burger joint that opened in 2012 could be closed by the end of this year or they could radically change their menu or discontinue their offer of free wi-fi. The shortcut that was good this year might be terrible road work infested territory next summer. (in that area of the country they don't mess about -- they completely remove the asphalt leaving only large pebbles around that threaten to punch holes through your car)

        In short: What has value today might be worthless next month. Someone might find my hamburger post (w/comments) worth a peek five years down the road, but I agree it is doubtful. Maybe I'll drive through there again some day and look at my posts from around that time to refresh my memory? Either way, why not store it in a sensible fashion?

    2. John Brown (no body) Silver badge
      Thumb Up

      Re: Just to be clear

      "The Director of Capacity Engineering at the largest social site... most of it is crap that no one will *ever* see or care about."

      My first though was "oh noes, another Gerald Ratner!"

      1. This post has been deleted by its author

        1. Michael Nidd

          @RedneckMother

          "sorry, but I STILL miss Gilda Ratner."

          Radner. Me too.

    3. Jamie Jones Silver badge

      Re: Just to be clear

      He didn't say it was crap - he actually said it was sad that this stuff was rarely - if ever - viewed.

    4. Bronek Kozicki Silver badge

      Re: Just to be clear

      He said nothing to be ashamed of, but I do not quite get what your point is.

      Unless you thought FB to be the gold standard of user generated content, in which case I do understand your disappointment.

  12. MnM

    Why not just

    not save it at all?

    1. Vociferous

      Re: Why not just

      It's a corollary to Murphy's Law: if you throw something away, you will need it later.

      1. Anonymous Coward
        Anonymous Coward

        Re: Why not just

        because 50 million plus users...

        1. John Brown (no body) Silver badge
          Coat

          Re: Why not just

          "because 50 million plus users..."

          ...suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened.

          Yes, the brown robe --------->

    2. Lusty Silver badge

      Re: Why not just

      "not save it at all?"

      Same reason you don't bin your family photo albums which are also pretty much write once, read never as he said. One day you'll look back and smile, but that may not be for 20 years by which time the album will at the very least be a bit dusty.

      1. DN4

        Re: Why not just

        "Same reason you don't bin your family photo albums"

        I do. If you keep them you might look back 20 years later and smile. But if you don't you will not miss them.

  13. Anonymous Coward
    Anonymous Coward

    Basically they want a CD-R type disk but lots of storage and faster. Don't we all.

  14. ops4096

    Surely WORN could save the N.S.A.'s Utah et al data barns heaps of dark money.

  15. Jolyon Smith

    Why not print the data in "2D bar code" form ?

    Then load into scanners when you need to read it ?

    This would also require that more tree's be planted to provide the pulp required to produce the storage medium (paper), consuming C02 in the process.

    By contrast, flash memory consumes irreplaceable raw materials and only ever contributes CO2 during manufacture and subsequent use.

  16. Anonymous Coward
    Anonymous Coward

    where did you put that laser?

    Flash doesnt really do. long term. but holographic storage well now that would be something.

    If only there were a company that specialized in holographic storage....

  17. ononeathome
    Unhappy

    Is this site only visited by teenagers and 20 something's?

    The ask of FB is stupid and no one calls them out?!? Instead people only sat they want it?!?

    This is what tape was invented for! With LTO now at 2.5 TB (yes TB) per cartridge (without compression), over 6TB with, no other form of long term storage data is going to come close to this any time soon. Yes tape is slow ... so what. FB itself says that people don't use it much. Tape systems (supported by hierarchical storage) are the exact solution to this. Density, cost and energy use (none while the tape sits there waiting for some to pull up grumpy cat).

    This industry just disappoints me more and more every day.

    1. ElNumbre
      Thumb Down

      Re: Is this site only visited by teenagers and 20 something's?

      Speed is the key for this use case scenario. Whilst tape is fine for bulk storage, its useless for giving up its data in a timely fashion. I'd imagine most FB users would want to only wait ~30 seconds to get that cat picture back. If FB adopted an 'archive to tape' model, perhaps along the lines of Amazon AWS Glacier where it can take 'some' hours to retrieve data, then firstly people would forget they'd requested the data, and secondly get slated for 'losing' information.

      For personal data backup, Amazon S3 then Glacier is perfect for me. Because it takes time to upload and then costs a nominal monthly cost to store, it forces me to be disciplined to only keep anything of value.

    2. dan1980

      Re: Is this site only visited by teenagers and 20 something's?

      Maybe the numbers don't support this, but I would think that from a power usage point of view, tape would be more costly. While it's not being used, flash can use no power as well but, unlike tape, power usage to retrieve something when needed is very low.

      Also, you'd need to manage the indexing and robotic tape libraries, etc... which adds complexity, if nothing else. I know that at volume that's pretty cheap but I get the impression that they are using 'conventional' disks and are okay with the price, they just want to see if flash can approach the price of normal drives if requirements for speed and read/write cycles are relaxed.

      The expectation is that at some point this data could be read as otherwise why keep it at all? Very little of this data WILL be read again but all of it MIGHT be read so if it is they still need to maintain some kind of reasonable latency. I can't see tapes really meeting that.

      Presumably they know their own requirements so it seems fair enough to me.

      Essentially, they are asking the question: How cheap can you make flash if we don't care about performance or re-write cycles?

  18. Mage Silver badge

    mv /dev/nul

    Or something, my UNIX is rusty. It's only Facebook. Do they care about any of it as long as it brings in advert revenue.

  19. Jess--

    I would go with a slightly different approach for a drive that would suit my needs perfectly...

    small section of high grade flash + massive section of prom (programmable read only memory)

    have the drive make any initial writes to flash and allow it to stay there for a period of time (30 days), anything that is unchanged over those 30 days move into prom (Write Once Read Many)

    this moving of data would be handled internally by the drive speed of transfer from flash to prom would not be an issue since the user would be unaware of the change and read access would be similar speeds regardless of whether the drive was pulling from prom or flash.

    It seems to me that the controllers for such a system already exist in the current generation of hybrid drives where commonly used files are held in flash with the remainder held on spinning platters, my suggestion would replace those platters with prom

    1. Phil E Succour
      Joke

      >> small section of high grade flash + massive section of prom

      I need to clean my glasses - I was quite disappointed when I realised I'd misread the last letter of that quote as a "n".

  20. ZanzibarRastapopulous

    Write Once Read Never

    Pretty much Facebook's entire business isn't it?

  21. Anonymous Coward
    Anonymous Coward

    Nah, just send it to the NSA

    They'll love to have it.

    Think of all those pictures of kitties and other inane objects they will have to wade through just in case one of them might have a child in it. Then they can go after you for distributing child porn. In second thoughts perhaps not but it would be a good way to DDOS the spooks, flood them with data. (Just an idea and in no way am I suggesting that it might possibly, sometime in the future happen ok)

  22. Justicesays
    Joke

    Just do it the way human memory does

    And essentially dedup.

    Gradually remove all those different pictures of beaches, or a kitten attacking a ball of wool, and replace them with just one iconic image!

    Eventually you will just have three pictures: a kitten, a dog and a naked lady on a disk somewhere.

  23. IGnatius T Foobar
    WTF?

    "Dense and cheap"

    "The Facebook ask of the industry is make the worst flash possible," Facebook's Jason Taylor told his keynote audience at the Flash Memory Summit on Tuesday in Santa Clara, California. "Just make it dense and cheap."

    Uncoincidentally, "dense and cheap" is a perfect description of the typical Facebook user.

  24. toughluck

    What about tape?

    Let's discuss the flash solution. It could be made in small or large modules, each would have its own advantages and drawbacks. Even though it's frugal, flash still needs power to function. The larger the basic module, the more power it will use. Then there's the matter of reliability. Larger modules would fail more frequently per module than small ones. And ultimately, they would be more expensive per byte because they would need more complex controllers. These factors would favor smaller modules. However, smaller modules would require more complex routing, switching, finally it would need very complex controllers per each brick of modules.

    Would it be cheaper than current flash technologies? Sure. Would it be cheap? Not by a long stretch. Flash is still 8-10 times more expensive than spinning drives. TLC doesn't bring the cost down far enough.

    It's also not a matter of density. At the same node, I suppose flash makers could make features denser, but even if they were twice as dense (which is rather unrealistic), we're looking at only four times the raw capacity -- which is still more expensive than spinning media.

    Interference would become a greater problem, and it would probably cause the usable capacity to not increase as fast as raw capacity did. Durability would suffer, of course, but as the guy said, it's not a problem for them, especially since they already don't delete the content, but keep it hidden.

    Nevertheless, it's still not a solution. Perhaps Facebook will be happy with the resultant module, even if it's expensive, if they think it will save power, or if it would be less complex to build and maintain, but I don't think so.

    --

    Which brings me to tape. There are T10000C drives that offer 5 TB per tape, and T10000D on the horizon which will offer more -- that's beyond the LTO roadmap at the moment, so I'm not talking about LTO. Tape has the nice property that when it's idle, it's not using up power and when a cartridge is needed, automation takes care of picking it up and mounting on a drive.

    --

    That said, I realize that if he said that waiting times for spinning up disks are too long, waiting half a minute or so to access a tape would probably be much too long for a user to wait. Caching part of the content on disk to wait until a tape is mounted would probably alleviate some of this concern. However, the service is free of charge, so Facebook pretty much has all power to set SLAs for it.

    1. Sandtreader

      Re: What about tape?

      Read-from-deep-archive times are the key here; what if your kitty pic is in the middle of a tape that's in a rack 10m from the drive? User Gets Bored, "FB sux".

      Flash doesn't need power when it's not being written or read either - I think dynamic power management would be front and centre of the storage device design.

      What I can't get my head round, though, is the implied contract of FB, Google/YouTube, Flickr an' all with their users is to keep all their data, however much they like, forever. Can they keep that up indefinitely? If not, who is going to prune it?

    2. Anonymous Coward
      Anonymous Coward

      Re: What about tape?

      So let me get this straight. Facebook has a perfectly good solution right now (racks of dormant hard drives, using essentially no power) but they want to improve access latency.

      Your suggestion is to switch to tape which will increase latency by an order of magnitude.

      Did I miss something?

      1. dan1980

        Re: What about tape?

        @AC 17:54

        No mate, you got it in one.

        Facebook are happy with the cost and power consumption of their existing solution and just want to find out if they can, for a similar price, use 'better' drives.

        Replacing large banks of magnetic hard drives with SSDs is presumably a SIGNIFICANTLY simpler task than replacing them with a tape-based solution because all the logic is unchanged - just swap the hardware. None of their processes would be affected at all.

        If they get what they are asking for, all that happens is they replace like-for-like hardware and almost instantly get better I/O, without having to implement another storage tier or additional caching; just new drives and faster access.

  25. Sarah Davis

    Or,..

    wouldn't another solution be simply not to store data after a user has deleted it - jus' sayin'

  26. bag o' spanners
    Devil

    A sensible retention schedule might shift their paradigm. No access requests for 5 years, bin it. If it was that important, the pea-brained user will have uploaded it from their own device, which they can then backup to their own choice of media, if they can avoid either breathing or chewing gum for long enough. The same issue affects every organisation that stores useless data, but doesn't have a sensible policy or strategy for binning it.

    4gb of 200kb photos (fb limit) is about 20,000 kittypix. Are people so technically inept that they can't burn a dvd of their timeless masterpieces to show to their fellow kittypix anoraks, in much the same way that holiday slides were employed to lethal effect in the olden times.. Who needs a will to live anyway?

  27. Anonymous Coward
    Anonymous Coward

    Should be easy enough

    SD usually cards seem to use crappy and cheap flash memory. Get whoever's making these chips to put them in an SSD for you, Facebook.

    I just checked on Newegg and the cheapest $/GB I could find for SD cards was 53 cents/GB, compared to 80 cents/GB for an SSD.

    Then again, hard drives are 5 cents/GB.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019