back to article Does Linux need a new file system? Ex-Google engineer thinks so

Former Googler Kent Overstreet has announced that a long-term project to craft a new Linux file system is at a point where he'd like other developers to pitch in. Since you're already asking why bother with a new file system, the answer Overstreet provides in this post to the kernel mailing list is that he wants to “match ext4 …

Page:

  1. This post has been deleted by its author

    1. thames

      Can you say that Hammer2 is any more complete, or even as complete, as Bcachefs? Bcachefs isn't just starting out. As the developer says, it has grown out of work done at Google over a number of years on caching data on SSDs. This announcement is simply to say that it is now becoming a stand alone file system rather than just a cache for other file systems.

      Overall, it sounds very interesting. It might eventually make a good replacement for ext4 on desktops and laptops. However, adoption of new file systems tends to be slow, because rarely change their file systems.

      P.S. - I wasn't the one who down-voted you by the way.

      1. Anonymous Coward
        Anonymous Coward

        "has grown out of work done at Google over a number of years ..."

        So it's Google's IP and so it will probably phone home to snitch on your content to them then?

        1. Robert Helpmann??
          Childcatcher

          Grown out of work done at Google

          That would explain the Googler's use of the term "metastasizing." It is refreshing when someone is up front about the fact their employer, past or present, is a cancer.

          1. nematoad

            Re: Grown out of work done at Google

            "“the bcache codebase has been evolving/metastasizing..."

            Aye, a poor choice of words, if he really wants this project to be taken seriously.

            Is it just me or are Google metastasizing into the new Microsoft?

            1. Anonymous Coward
              Joke

              Re: Grown out of work done at Google

              Flip that statement around and it still seems true. We will have found the Antichrist when they merge.

        2. oldcoder

          nope.

          Being part of the kernel makes it GPL... any "phone home" attempts would be shown up, VERY loudly... and removed.

    2. oldtaku Silver badge
      Go

      "On the other hand, why not just put the effort behind the development and porting to Linux of the Hammer2 filesystem?"

      Practically, it'd be more productive if everyone got together and worked in perfect harmony on the UberFileSystem. But helping debug some else's already 3 years overdue file system isn't nearly as fun as writing your own new one. And it's the *nix way to have 5 projects when you could have 1. I can't blame him.

      1. thames

        @oldtaku - "Practically, it'd be more productive if everyone got together and worked in perfect harmony on the UberFileSystem."

        Well, there's also the thing that there's no one size fits all file system either. File systems that work best on large file servers can be excessively resource hungry on PCs. File systems that work best on PCs don't have all the advanced features or performance for large file servers. Systems like ZFS or BTRFS tend to take up to take huge gobs of RAM to run properly and that's probably inherent to the features which people want to use them in.

        The article and the original post don't say is what applications they are targeting with Bcachfs. What I would like to see is a replacement for Ext4 before that file system runs out of steam due to increasing disk size. Things like checksumming are important with large disks because of how long it takes to run a conventional scan on them. Right now the plan seems to be for BTRFS to replace Ext4, but I'm not sure that's the right decision.

        1. Adam Inistrator

          btrfs memory requirements

          1. "... BTRFS tend to take up to take huge gobs of RAM to run properly ... "

          2. "... Right now the plan seems to be for BTRFS to replace Ext4."

          something contradictory in these statements. I think 1 is false

      2. Dinsdale247

        FreeBSD

        "Practically, it'd be more productive if everyone got together and worked in perfect harmony on the UberFileSystem. But helping debug some else's already 3 years overdue file system isn't nearly as fun as writing your own new one. And it's the *nix way to have 5 projects when you could have 1. I can't blame him."

        No, that's the GNU/Linux way of doing things. That's why GNU/Linux has 5 projects and FreeBSD has zfs.

      3. Dr. Mouse

        But helping debug some else's already 3 years overdue file system isn't nearly as fun as writing your own new one. And it's the *nix way to have 5 projects when you could have 1. I can't blame him.

        This is the software equivalent of https://xkcd.com/927/

        Also, I have in the recent past developed a new feature for existing software. The amount of work it took just to understand where it would fit in almost pushed me to rewrite from scratch. In the end I shoehorned the software in, but only for my own use as making it all fit with the project's guidelines was too much of a ball ache. It did the job I wanted, but writing my own from scratch would have been no more difficult and far more fun (at least for my own use). It would have been more restricted in it's use, but it would have done the job I needed at the time.

        It's always the way, I find. Learning how to develop for an existing codebase is a lot of hard work. Once you know, great.

      4. Gary Bickford

        Multiple projects can be a good thing

        Back in the heyday of Japanese 'Asian Tiger" when their cars were eating everyone's lunch, one of the methods they were using for development of new products (or so I read at the time) was to assign the same project to two or three independent teams, who each competed to come up with the best new design. At some point either a winning team was selected, or the best points of all of them were merged and the most successful team was given the lead to finish.

        This seems more expensive, but it probably greatly reduces the probability of abject failure so is probably cheaper and almost certainly better in the long run.

        1. Number6

          Re: Multiple projects can be a good thing

          That's how venture capitalists work - fund ten projects in the expectation that most of them will fail, only they don't know at the start which one or two will be successful so they have to start with all of them.

    3. bazza Silver badge

      integrated filesystem checksumming, for example, should really be everywhere.

      Well in effect it has been for a long time, though not necessarily at the level of the file system. Physical storage has had error detection / correction for a long time now.

      It was only with the advent of very large storage devices that their on board ECC became inadequate for "ensuring" (there's no such thing as a guarantee) data accuracy. That's led to file systems like ZFS putting in an extra layer of ECC of their own to compensate.

      Incidentally I think the characteristics of the ECC in ZFS were carefully chosen to accommodate the typical bit error rate achieved by HDDs. Getting that right in a file system design is important; just slapping in a CRC something-or-other makes no sense unless one matches it's parameters to the BER of the underlying physical devices. Too much in the file system and you're wasting space and throughput, too little and the BER might be higher than desired. Of course, choosing the BER that's right for the business is another matter...

    4. Jim 59

      We've all needed a new file system since about 1992, viz some kind of tagged file system that frees us from the directory structure, while still somehow preserving it. Something equivalent to a nosql database. But this is still a vague fantasy.

      In the real world, I'd like ZFS-style deduping please, and optimisation for both flash and rotating drives. The deduping alone would allow us to bin 99% of the worlds disks. Maybe that's why it hasn't taken off in the way everyone expected.

      1. Preston Munchensonton
        Pint

        We've all needed a new file system since about 1992, viz some kind of tagged file system that frees us from the directory structure, while still somehow preserving it. Something equivalent to a nosql database. But this is still a vague fantasy.

        In the real world, I'd like ZFS-style deduping please, and optimisation for both flash and rotating drives. The deduping alone would allow us to bin 99% of the worlds disks. Maybe that's why it hasn't taken off in the way everyone expected.

        Damn it, Jim! You've had 27 years. Why isn't this done yet?

    5. Anonymous Coward
      Anonymous Coward

      "On the other hand, why not just put the effort behind the development and porting to Linux of the Hammer2 filesystem?"

      Hammer2 is a great file system, but wouldn't there be licensing issues? Would the GPL nutters try and shackle it with GPL v3 licenses?

  2. oldtaku Silver badge
    Thumb Up

    Sure, why not.

    ... as long as we can default to ext4 and pick others based on our level of devil-may-careness. And as long as you don't kill your wife.

    1. Anonymous Coward
      Anonymous Coward

      Re: Sure, why not.

      But what if she's _really_ annoying?

      1. John H Woods Silver badge

        Re: Sure, why not.

        "But what if she's _really_ annoying?" --AC

        Or FAT?

    2. Anonymous Coward
      Anonymous Coward

      Re: Sure, why not.

      "And as long as you don't kill your wife."

      Cold, man. really cold...

  3. Anonymous Coward
    Anonymous Coward

    Ah, so it's like ZFS but unfinished.

    Let's reinvent the wheel!

    1. Ian Michael Gumby
      Devil

      Re: Ah, so it's like ZFS but unfinished.

      Thats the silicone valley way!

      1. Anonymous Coward
        Holmes

        Re: Ah, so it's like ZFS but unfinished.

        >"silicone valley"

        Been dreaming about your mother recently?

    2. bri

      @ Martijn Otto - You mean btrfs, surely

      Did Oracle release ZFS under GPL in full? As far as I am concerned, ZFS doesn't exist as it's not supported in any major Linux distro. And if I wanted something proprietary, I'd go GPFS (I prefer IBM to Oracle).

      Anyway, btrfs still doesn't feel production ready, so bcachefs may be interesting option (license permitting).

      1. Six_Degrees

        Re: @ Martijn Otto - You mean btrfs, surely

        "Purity of Essence. EOP. OPE. It's one of those!"

        1. bri

          @Six_Degrees - You mean btrfs, surely

          Oh come on. Even Oracle Linux doesn't come with ZFS as supported standard. And they should be able to ship and support it.

          One thing is hobby, where ZFS may be gold, you can bolt it on Fedora or whatever, other thing is production, where only supported configurations or at least widely used stable combinations with major distros count. ZFS on Linux is neither (sorry but home & hobby doesn't count).

          1. Alan Brown Silver badge

            Re: @Six_Degrees - You mean btrfs, surely

            "Even Oracle Linux doesn't come with ZFS as supported standard."

            Of course not. If it did they wouldn't be able to charge you 6 figure support fees for ZFS on Solaris.

            1. admiraljkb

              Re: @Six_Degrees - You mean btrfs, surely

              @Alan Brown

              ""Even Oracle Linux doesn't come with ZFS as supported standard.""

              "Of course not. If it did they wouldn't be able to charge you 6 figure support fees for ZFS on Solaris."

              +1 for Oracle and crazy support fees, but in this case, BTRFS is their green field replacement for ZFS. It started off that way before the Sun acquisition, but continued afterwards. Hence ZFS doesn't come as an official option with Oracle Linux as its considered a bit "legacy" on top of the maneuvering needed for licensing compatibility issues... I wish they'd push more resources onto BTRFS and finish off the last 10% of whats needed.

              1. phil dude
                Thumb Up

                Re: @Six_Degrees - You mean btrfs, surely

                I agree. I am a zfsonlinux user (it is amazing and I am a *minor* kernel contributor -solved a problem I had, of course!).

                BTRFS is clearly the future as it will be in *every* kernel. Rather than ext4 which is , well, in need of a replacement...

                Remember, once you have COW, checksums and snapshots per volume, you have sufficient flexibility to manage many different distributions on the same machine.

                P.

      2. Antonymous Coward
        Holmes

        Re: @ Martijn Otto - You mean btrfs, surely

        BCacheFS is hardly "production ready" either. Its advertised performance is FAR from reality too.

        1. bri

          @AC

          Where do I say that it is? OTOH it still *may* achieve production-level readiness faster. 'May' is the word, but until either of them is production ready, no one knows which is ultimately better way. So it is beneficial to pursue development of both of them.

          And that was my point, which is not shared by the initial poster in this thread.

      3. Arthur the cat Silver badge

        Re: @ Martijn Otto - You mean btrfs, surely

        "Did Oracle release ZFS under GPL in full?"

        No, Sun released it under CDDL, a variant of MPL. Perfectly free and usable by everybody. It's only rms' desire to make GPL The One True Licence To Be Obeyed By All that prevents its integration with Linux.

        Once you've seriously used ZFS (and recovered from dead disks with no data loss) you wouldn't want any other file system.

        1. Microchip

          Re: @ Martijn Otto - You mean btrfs, surely

          zfsonlinux.org has a kernel module, but as mentioned in other comments, non-GPL compatible. Fast though, seemed to work well when I was testing as an alternative to FreeNAS with ZFS. I've zero issue throwing a non-GPL module in, but purist may be unhappy with the idea.

          1. Bronek Kozicki

            Re: @ Martijn Otto - You mean btrfs, surely

            purist GPL zealots may be unhappy with the idea

            FTFY, because CDDL is open source. Unfortunately because it's not GPL, it cannot be included in kernel source tree - but it can be (and has to) built when deploying your own kernel. There are source (as opposed to binary) packages included in major distributions just for this.

        2. Anonymous Coward
          Anonymous Coward

          Re: @ Martijn Otto - You mean btrfs, surely

          > No, Sun released it under CDDL, a variant of MPL. Perfectly free and usable by everybody.

          And indeed, you can even sell commercial appliances which use ZFS without having to release the source code. This is a much more open and free licence than the GPL.

          It seems unlikely Oracle chose this licence to prevent ZFS being used with Linux, when by releasing under GPL they could have maintained a revenue stream by licensing the code to commercial users.

        3. John Hughes

          Re: @ Martijn Otto - You mean btrfs, surely

          "perfectly free and usable by everyone"?

          No, the CDDL was explicitly written to be incompatible with the GPL.

      4. Richard_L
        Unhappy

        Re: @ Martijn Otto - You mean btrfs, surely

        I just wish Oracle would change the licencing of ZFS out so it can be included with distros by default, instead of being cast out into a legal wilderness as it is now. Then we could take BTRFS off to the woodshed and put it out of its misery and not have to wait years for this bcachefs to be ready.

        I've been using BTRFS as my filesystem on SUSE and it's a pain in the arse. SUSE recommend it as the default filesystem for root, yet it has a nasty habit of filling up and silently refusing to write any more data to the disk even though standard utilities like df continue to report plenty of free space.

        Neither SUSE nor BTRFS appear to monitor for the approach of this condition nor alert the user or administrator to it when it happens. SUSE have a tool that merrily snapshots the **** out of any btrfs filesystem, but will they monitor that same fileystem for chronic constipation? Nope... go and write your own script to periodically rebalance your chunks. Grr.

        1. bazza Silver badge

          Re: @ Martijn Otto - You mean btrfs, surely

          I just wish Oracle would change the licencing of ZFS out so it can be included with distros by default, instead of being cast out into a legal wilderness as it is now.

          Well, it's up to them I suppose. It's their code, and I think everyone is grateful that they chose to share it at all. They obviously had specific goals on control and re-use that they felt GPL wouldn't achieve, so wrote a license to suit. It's not Sun/Oracle's fault that Linux is under GPL2. We can make do and mend with building our own kernel modules or getting some pre built ones.

          FreeBSD has had no trouble at all adopting ZFS. There's OSX implementations, and reportedly MS briefly considered putting it into Windows. Rigid and unwavering adherence to the current GPL2 guarantees that Linux is always going to be hampered this way, which ultimately is not beneficial for the Linux community.

          1. HPCJohn

            Re: @ Martijn Otto - You mean btrfs, surely

            I agree.

            I am a fan of GLusterfs, and it just seems such a natural good fit with ZFS - ZFS filesystems on the bricks, then scale out your filesystem, replicate etc. using Gluster.

            You can easily do this yourself with zfsonlinux.

            However I'm told that Redhat can't roll this out as a supported configuration with Redhat storage, because of this license restriction. Arggh...

          2. future research

            Re: @ Martijn Otto - You mean btrfs, surely

            Sun released the code under the CDDL, and FreeBSD adopted it. My real problem is that Oracle then stopped releasing any updates, so the OpenZFS has had to create a fork from the last Sun release of opensolaris. This fork will therefore be incompatible with the later version of Oracle ZFS.

        2. thames

          Re: @ Martijn Otto - You mean btrfs, surely

          @Richard_L - The CDDL license used by ZFS was carefully crafted to make it incompatible with the license used by Linux. Sun was trying to establish OpenSolaris as a major open-source OS, and didn't want people to simply take the best bits of it (basically ZFS and Dtrace) and stick them into Linux. That would have killed OpenSolaris as there would have been no reason for most people to use it.

          http://www.cnet.com/news/sun-open-source-license-could-mean-solaris-linux-barrier/

          "The CDDL is not expected to be compatible with the GPL, since it contains requirements that are not in the GPL," Claire Giordano of Sun's CDDL team said in its submission. "Thus, it is likely that files released under the CDDL will not be able to be combined with files released under the GPL to create a larger program."

          ZFS on BSD was likely not considered a problem, because the user base of BSD is so much smaller.

          1. bazza Silver badge

            Re:Thames

            The CDDL license used by ZFS was carefully crafted to make it incompatible with the license used by Linux.

            Isn't the one defining point of GPL that any other license, no matter what it says, is essentially incompatible with it?

            1. phil dude
              WTF?

              Re: Re:Thames

              No. The goal of the GPL was that software that was free could stay free, and not become locked down by accumulated changes.

              The reason why parasitic companies (like Oracle) do what they do is because the ambiguity they introduce with a "special" license, makes it difficult for other businesses to count on it. Look at all the problems Google is having...

              Hence we are in the insane situation that the source code, currently running on *your* machine, could become illegal to possess due to a court decision. If you are at home, who cares. But corporations would become liable and that FUD changes the market, so that inferior products become entrenched through this proxy extortion. (look up FAT on android and Microsoft's own version of this anti-consumer trend)

              RMS might have seemed extreme a few years ago, but he was spot on.

              P.

            2. thames

              Re: Re:Thames

              @bazza - "Isn't the one defining point of GPL that any other license, no matter what it says, is essentially incompatible with it?"

              Er, no. There is non-GPL code in the Linux kernel, for example some MIT stuff. However, one of the major characteristics of any GPL-type license is the "no additional restrictions" clause. That is, you may not impose any additional licensing restrictions or requirements on the software which were not originally present. That is intended to keep the software open.

              The reason that CDDL is incompatible with GPL is exactly what the Sun CDDL license drafter said in on of my earlier posts.

              "The CDDL is not expected to be compatible with the GPL, since it contains requirements that are not in the GPL," Claire Giordano of Sun's CDDL team"

              As you can see, according to the licensing experts at Sun, CDDL would impose additional licensing requirements which are not present in GPL.

              There's a list at the following link which tells you which common FOSS licenses are compatible with GPL.

              http://www.gnu.org/licenses/license-list.en.html#GPLCompatibleLicenses

              Here's a few compatible examples from their list:

              * Artistic License 2.0 (Perl).

              * Berkeley DB License (Sleepycat Software license).

              * Boost Software license.

              * Newer BSD (2-clause BSD - very early versions had an third clause with a mandatory advertising requirement which was not compatible and which caused loads of headaches for other people as well).

              * MIT (basically the same as 2 clause BSD).

              * Intel Open Source License.

              * MPL - Mozilla Public License.

              * Public Domain.

              * Python 2.1.1 or newer.

              * Ruby license.

              * X11 License.

              * etc.

              There's more, but I've just picked the more common ones.

              In addition, GPLV3 is compatible with Apache 2.0, but not the older GPLV2 license which the Linux kernel still uses. One of the big reasons for updating the GPL to version 3 was to make it compatible with the Apache license. GNU recommends Apache 2.0 over a BSD/MIT style license because it deals with patent issues (which GPLV3 also addressed in its own update).

              The things that usually tend to make licenses incompatible with GPL are those which have more restrictions. For example, the Eclipse License says that the EPL is governed by the laws of New York. The US isn't the whole world and you're not allowed to impose a restriction like that on GPL software. A lot of the "corporate" open source licenses have clauses like that, which quite frankly makes them pretty useless to anyone other than the original author. There are other explanations at the above web site as well.

              Another good example is the original JSON license had a clause which said: “The Software shall be used for Good, not Evil.” That made it non-GPL compatible because you have the right under the GPL to use the software for evil if that is what you want. That might sound like nit-picking, but these are the sorts of details which compatibility can fall down over.

              However the big licenses today are GPL, MIT (BSD), Apache, and MPL. MIT and MPL are compatible with versions 2 and 3 of the GPL, and Apache is compatible with version 3 of the GPL.

              So yes, CDDL is not compatible with GPL, but it's not because there is anything exceptional about the GPL itself.

          2. oldcoder

            Re: @ Martijn Otto - You mean btrfs, surely

            BSD also allows them to take any improvements and make them proprietary...

            GPL does not.

        3. Anonymous Coward
          Anonymous Coward

          Re: @ Martijn Otto - You mean btrfs, surely

          > I just wish Oracle would change the licencing of ZFS out so it can be included with distros by default,

          Sod off!

          At the most, maybe a dual-licensing setup. Many are grateful that the legalistic cancerous GPL hasn't been used.

          Here's a thought: Why not instead wish your system wasn't licensed under such a restrictive environment, rather than have the arrogance to expect others to change their code to fit your narrow-world license?

          1. Adair Silver badge

            Re: @ Martijn Otto - You mean btrfs, surely

            In reply to: 'Here's a thought: Why not instead wish your system wasn't licensed under such a restrictive environment, rather than have the arrogance to expect others to change their code to fit your narrow-world license?'

            Quit your whining. You're free to pick whatever licence suits you philosophically and economically. If someone else doesn't like the licence you have chosen, well they're always free to go away and do the work you have done and give the result a licence that suits them.

            The GPLs suit certain purposes, they're not supposed to be a panacea.

            1. Anonymous Coward
              Anonymous Coward

              Re: @ Martijn Otto - You mean btrfs, surely

              > In reply to: 'Here's a thought: Why not instead wish your system wasn't licensed under such a restrictive environment, rather than have the arrogance to expect others to change their code to fit your narrow-world license?'

              > Quit your whining. You're free to pick whatever licence suits you philosophically and economically. If someone else doesn't like the licence you have chosen, well they're always free to go away and do the work you have done and give the result a licence that suits them.

              Funnily enough, your reply reads more like an agreement with the post you are replying to. It's the PREVIOUS poster that was whining about the license being chosen.

              I assume English isn't your first language?

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like