back to article SMASH the Bash bug! Apple and Red Hat scramble for patch batches

A fresh dump of Shellshock patches were released on Friday night in the latest move to stamp out the Bash shell security vuln that has the potential to blight millions of Linux, Unix and Mac OS X machines. Red Hat said in a blog post that the threat from Shellshock was receding now that patches had been issued for most …

  1. Snowy Silver badge

    I wonder

    How many researchers are going to look at other quite obscure features that are rarely used for other security vulnerabilities, I know if I was one I would be.

    1. Anonymous Coward
      Anonymous Coward

      Re: I wonder

      I imagine the miscreants are already. With heartbleed and now this having been present for 22 years, I suspect there is going to be quite a bit more attention on a lot of software previously assumed to be safe by those with both good and bad intentions.

    2. Anonymous Coward
      Anonymous Coward

      Re: I wonder

      What a mess. No wonder Windows Server is now the most popular web hosting platform.

      1. Solmyr ibn Wali Barad

        Re: I wonder

        Careful now. Should you ever bump into Eadon, you'd create a sizeable crater, lots of gamma radiation, and possibly a shear in the space-time continuum.

        Have an upvote, though. For...erm...amusantly relentless reflection of the vividly virtual alternate reality.

      2. handledadog

        Re: I wonder

        At least with Linux, the improvements stay, never to be seen again. With windows, every new version is a major re-design of their pathetic code base.. That's why with Windows you see the same security bugs over and over.

  2. El Andy

    "When a second issue with Bash was found a few minutes after the first one went public, we knew there was something wrong. We could have followed a duct-tape approach and issued patches to our customers quickly or we could have done this correctly"

    Perhaps the more pertinent question is why they didn't bother to do the first fix "correctly"

    1. Flocke Kroes Silver badge

      Because the flaws were very different

      The first flaw was that when bash imported a valid function from the environment, it interpreted anything after the function as a bash script.

      The second flaw was if the function definition in an environment variable started out correct, but contained a certain type of invalid syntax in the middle, followed by the character '>' which redirects stdout to a file and a \ at the end to say that the rest of the command is 'on the next line of input', then bash would keep the '>' and put it at the start of the next line of text to be interpreted. Normally, the first word in a line of bash is a command, followed by its arguments. Redirecting stdout can be placed anywhere, and the '>' and file name are removed from the text so they do not show up as an argument for the command.

      That second flaw is a radically different path through the code that handles an odd corner case. It is not surprising that people concentrating on fixing a different problem while keeping as much as possible of the interpreter the same (to avoid breaking any bash scripts) missed this.

      Bash collected handy features because they were useful on the command line. Years ago, sh was often a link to bash so those features would be available to all the scripts in the operating system, and would be available when one command starts another with the 'system' C library function. All those handy features created a large attack surface, which was dealt with in multiple ways:

      The 'system' library function became unfashionable. Programmers should use something like 'execve' instead, which does not invoke 'sh'. The link from sh to bash changed to point at a cut down shell like ash. Bash could continue to grow handy features, but ash remained small and easier to audit for security issues. Part of the reason bash had a major flaw for decades was that people were looking at ash and its derivatives instead. Security researchers did not expect bash to be used where security was required.

      1. Nigel The Pigeon

        Re: Because the flaws were very different

        What a great and informative comment. It is better than most of the articles about the bug.

        Not sure why anyone would downvote it?

        1. Tim Roberts 1

          Re: Because the flaws were very different

          downvote was from someone who either didn't understand the explanation, or is clearly so smart that he didn't need the explanation, or knows better and is too busy to write his own explanation.....

          1. Anonymous Coward
            Anonymous Coward

            Re: Because the flaws were very different

            I've noticed it is very difficult to collect dozens of upvotes without one or two random downvotes, even for something that is totally uncontroversial and useful like that post. Some people just hate success I guess, and feel better about themselves by swimming against the tide via downvoting something any right thinking person know deserves an upvote (or at worst, the lack of an upvote)

            If someone had a legitimate issue with the content, anyone taking the time to downvote it would have taken the extra 30 seconds to add a one line post with the correction.

            1. Anonymous Coward
              Anonymous Coward

              Re: Because the flaws were very different

              Wintrolls downvote anything that is positive about linux.

              That's their job.

              1. Anonymous Coward
                Anonymous Coward

                Re: Because the flaws were very different

                While Linux worshipper do their best to dismiss any issue - even if they are big as this....

            2. ProperDave
              Coat

              Re: Because the flaws were very different

              I've had people downvote my occasional attempts at innocent humour on some topics.

              It would make more sense if downvoters would stop to share their concerns with a post - though in situations where the OP is being unreasonable, then it probably doesn't need a comment, but then that could depend on someone's perception of 'unreasonable'.

              Back on topic though, this seemed an excellent explanation - I've been frustrated with all the news articles on the exploits as the majority have been all 'end-is-nigh' panic fests, without actually providing any insight into the actual problem, I feel I now understand the issue a little better, though I still don't fully comprehend the attack vector - are we talking naughty perl/cgi scripting? Surely the only risks here are if a dodgey script is put in an executable position on the server? I clearly need to do more reading :o

          2. Anonymous Coward
            Anonymous Coward

            Re: Because the flaws were very different

            Look for 'excusatio non petita, accusatio manifesta'

            Of course the long explanation was just a way to hide the fact the whole vuln management has been dreadful.

            1. Maventi

              Re: Because the flaws were very different

              "Of course the long explanation was just a way to hide the fact the whole vuln management has been dreadful."

              Sorry, what? It was a pretty useful and informed explanation! Where's yours?

              1. Anonymous Coward
                Anonymous Coward

                Re: Because the flaws were very different

                Mine? Exactly "the vulnerabilty management was dreadful". They rushed out a patch and the same type of attack vector was still exploitable. Because they just put a lame workaround insted of really reviewing the code and re-writing it correctly. Sure, you can blah-blah about tech details in an attempt to hide it behind them - any botched patch has a "rationale", some "corner case", etc. - but it doesn't change the fact that the vuln management has been dreadful, and let's hope this is the final patch and we'll not have to upgrade systems again and again.

                Also releasing it Friday night made a lot of sysadmin happy, that night and the following Saturday, believe me.

                1. Anonymous Bullard

                  Re: Because the flaws were very different

                  "the vulnerabilty management was dreadful"

                  How would you have done it better?

                2. Maventi

                  Re: Because the flaws were very different

                  "Also releasing it Friday night made a lot of sysadmin happy, that night and the following Saturday, believe me."

                  So rather than get a hole patched as soon as possible, you would prefer to wait until next Tuesday?

                  I know a lot of sysadmins (and am one myself) and we are all much more concerned about the severity of the vulnerability rather than patching it as the latter is was a pretty trivial process. Sure it wasn't convenient, but it's always much better to have something ready immediately than risk cleaning up after a huge exploit.

                  I still fail to see any justification for downvoting what was simply a good explanation of the problem.

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Because the flaws were very different

                    (different AC here)

                    Sure it wasn't convenient, but it's always much better to have something ready immediately than risk cleaning up after a huge exploit.

                    Exactly!

                    Personally, I would much prefer to get a patch ASAP than wait until a Tuesday. But it's ok, you don't have to patch as soon as you get it.

                    What would you like to do on Monday morning? Explain to your boss/clients that your systems have been compromised because you couldn't be arsed logging in over the weekend? then begin the ordeal of rolling back off-site backups (losing a day's work if your backup process is very good), re-setting passwords, new private keys, perhaps dealing with Police, having your security practices and recent activities audited, etc.

                    Could you imagine the discussion of the package maintainers being something like:

                    : "phew, our patch is ready to release!"

                    : "Oh no.. let's hold it until Monday, we don't want to bother the sys-admins on a Friday afternoon!"

                    : "Let's make it Tuesday, no body likes Mondays"

                    1. Billa Bong

                      Re: Because the flaws were very different

                      "Personally, I would much prefer to get a patch ASAP than wait until a Tuesday."

                      Do you imagine the developers sitting there are deciding to release a patch next week just because they couldn't be bothered to produce one today? I personally thought that their arguments were well reasoned given the circumstances. They have to trade off getting a complete patch out on Tuesday vs. missing Tuesday because of the coordination in getting out a patch on Saturday, Sunday and Monday as well.

                      On at 100+ deployment I would personally put in place mitigation factors immediately while I wait for a complete patch to be provided, then roll out the patch. I'm not particularly enthusiastic about extending a 2 touch approach that is 100% effective to 5 touch because a new patch has been issued on each successive day.

                      I appreciate that for some mitigation factors won't be appropriate or implementable, but those people would presumably be shutting *everything* down until the all clear has been sounded rather than running a system that may or may not have an effective patch.

                  2. Anonymous Coward
                    Anonymous Coward

                    Re: Because the flaws were very different

                    No, be smart, disclose it on a Friday night, so a lot of systems that aren't managed 24x7 risk to be left unpatched for two days... that's the way to manage security issues. Do you believe that Microsoft choose Tuesday and not Friday night "just because"?

                    Remember that it's the disclosure that makes a vulnerability more dangerous - have you ever seen a vulnerabilty risk curve?

                    I just said it made a lot of sysadmin "unhappy" - do you like working on Friday nights, Saturdays or Sundays - especially when you thought delievered patches have already fixed the issue fully?

                    Patching a trivial process? On your bedroom server, maybe - production systems needs a full test on a preproduction one before applying *any patch*. There is no "trivial patch" - and the management of this one shown it fully. And having to patch and re-patch is not funny at all.

                    Or is not a problem of yours because you have not such a job?

                    Again, you fail to notice that that long explanation was only a smoke curtain to hide the fact that there has been a patch, than five more vulnerabilities were found - sure, it's fine to discuss them - but it's still a big mismanagement of the whole problem by the Bash maintainers - they have a broken parser and thought some workarounds could be enough - when clearly they weren't.

                    But of course our Linux community is more interested in denying any issue and mismanagement (that's Windows only, right? can't affect us, right?), instead of trying to really learn from it and put in place better safeguards and better vullnerability and patch management.

                    1. Rick Giles
                      Linux

                      Re: Because the flaws were very different

                      ""No, be smart, disclose it on a Friday night, so a lot of systems that aren't managed 24x7 risk to be left unpatched for two days"

                      What two days do you speak of? Saturday and Sunday? You get those off? Oh, you must be one of those 8 - 5, M - F admins that I hear about occasionally.

                3. SteveK

                  Re: Because the flaws were very different

                  Also releasing it Friday night made a lot of sysadmin happy, that night and the following Saturday, believe me.

                  It's not Friday night everywhere at once...

                4. elip

                  Re: Because the flaws were very different

                  Sorry to spoil your hopeful nature...Michal Zalewski found at least one other exploitable bug related to env parsing...get ready to patch again very soon.

        2. Technogeek

          Re: Because the flaws were very different

          The person probably didn't have their glasses on and hit the wrong vote arrow, either that or they just don't understand sensible and enlightened comments.

        3. Tom 38

          Re: Because the flaws were very different

          Not sure why anyone would downvote it?

          There were several things that are factually incorrect, system(3) has always been uncool for many many reasons, '/bin/sh' was a long time ago "the" bourne shell, not a symlink to bash (which is a Linux-ism, no other UNIX does that), several others. If you are a factual pedant, you might downvote for those reasons.

          (I'm not the downvoter!)

      2. Anonymous Coward
        Anonymous Coward

        Re: Because the flaws were very different

        It makes a lot of sense to use ash or dash or another simpler POSIX sh implementation for the system script shell rather than bash. Yet RHEL and Suse Enterprise both link /bin/sh -> bash. For one, the Ubuntu folk were right.

        1. John Sager

          Re: Because the flaws were very different

          There are quite a few scripts in /bin, /sbin, /usr/bin & /usr/sbin that are bash scripts on Ubuntu though. I looked at one or two and it didn't look like they used specific bashisms. The biggest offenders were scripts associated with gzip.

          1. thames

            Re: Because the flaws were very different

            Normal bash scripts aren't in themselves a problem, so the ones that John Sager mentions shouldn't be an issue. A lot of people will specify #!/bin/bash because their default shell is bash, and they don't want to test their script with every possible shell. They have bash, it works with bash, and so they specify bash.

            I think the main reason that Ubuntu uses dash instead of bash (although bash is still included) is because they inherited it from Debian (Ubuntu is a Debian derivative). If I recall correctly, Debian used dash because they are still using System V init, and dash started up quicker than bash and Debian were looking for faster boot up times (System V init starts up many copies of the shell). Ubuntu uses Upstart instead of System V, but from their point of view, there would be no reason to change Debian's choice of default shell.

            1. Gunnar Wolf
              Linux

              Re: Because the flaws were very different

              If I recall correctly, Debian used dash because they are still using System V init, and dash started up quicker than bash and Debian were looking for faster boot up times

              It is one of many reasons. But if we were to find one single reason, I'm sure we'd end up finding my friend Raphael Geissert sitting on it: He was quite active for some time finding and explaining bashisms, different constructs which are handy but break strict-POSIX shell programming. Allowing the system to properly run with a strict POSIX shell benefits low-disk-space systems (i.e. the embedded world), allows for faster start-up times, and leads to less head-scratching due to hard to identify bugs.

              If I recall correctly, the drive to migrate the default shell from bash to dash was in 2009, somewhat before the migration to Upstart/SystemD/OpenRC/AnyOtherInitStyle.

              - Just One More Debian Developer ;-)

      3. thames

        Re: Because the flaws were very different

        Most people doing things like PHP CGI aren't using most of the features of even the simplest standard shell. I have to wonder if the solution would be a minimalist shell substitute that Apache or Nginx (or whatever) called instead of the default shell and which provided just the features needed.

        Gluing two complex bits of software together provides a lot of scope for things to go wrong, especially when one of them (bash) was designed a long time ago.

        1. Tom 38
          WTF?

          Re: Because the flaws were very different

          Most people doing things like PHP CGI aren't using most of the features of even the simplest standard shell. I have to wonder if the solution would be a minimalist shell substitute that Apache or Nginx (or whatever) called instead of the default shell and which provided just the features needed.

          lolwut?

          CGI does not use shells.

          Apache/Nginx/whatever do not invoke a shell to invoke a CGI app

          The only CGI applications which utilize shells are CGI scripts written in a shell language

        2. Anonymous Coward
          Anonymous Coward

          Re: Because the flaws were very different

          "I have to wonder if the solution would be a minimalist shell substitute that Apache or Nginx (or whatever) called instead of the default shell and which provided just the features needed."

          You mean you're not doing this already? (Also, setting the web user shell to /bin/nologin)

          Looking at our webserver logs, at least some of the exploits attempted were effective against the cgi-bin interpreter, but didn't run because the webserver doesn't have a shell (something which can be worked around with the right cgi-bin call, sigh....)

          The most prolific attacks are skiddies trying to install IRC bots but I did see a few attempts to install banking trojans and C&C software.

          There's a LOT to be said for sandboxing webservers and firewalling every bit of unnecessary OUTBOUND traffic along with the usual inbound filters.

        3. handledadog

          Re: Because the flaws were very different

          PHP isn't CGI.

      4. MacroRodent

        Re: Because the flaws were very different

        The best solution would be to kill the function inheriting feature entirely. Other shells do not have it. Should try sometime what happens if I disable it, I suspect RHEL would still work fine.

      5. Anonymous Bullard

        Re: Because the flaws were very different

        I'm curious, though.. why is it a flaw of bash? - I'd call it a feature.

        The flaw is that the externally facing server software is, in one way or another, directing requests through software that is designed to control the system.

        I've been using bash for years, and I'm still learning new features. How can something like that be used to process external requests?

        (I'm not the phantom down-voter, by the way)

      6. Lance the Boil

        Re: Because the flaws were very different

        I keep seeing references saying that /bin/sh is not commonly linked to bash. But in Redhat Server it is. How can the (arguably) most common distribution (if you include CentOS) not be mentioned as an exception if it is?

        Just asking as I've always seen bash as linked to /bin/sh

  3. Clockwork

    Bullet dodged

    Patched 16 hours before the first logged attempted attack on my box.

    1. Anonymous Coward
      Anonymous Coward

      Re: Bullet dodged

      several attempts logged on mine, because its not latest debian release.

      Fortunately I dont use cgi-bin, so its empty and anyway debian uses dash by default..

      I always thought bash was a load of crud anyway, from the name.

    2. Lance the Boil

      Re: Bullet dodged

      "Patched 16 hours before the first logged attempted attack on my box."

      Note the use of the work 'logged'. The bug has been around for decades. Do your logs go back as far the server build time?

      What worries me is the number of exploited servers before the announcement. It will be less, but someone found the bug and announced it. What's the guarantee that he was the first to find it?

      1. Alan Brown Silver badge

        Re: Bullet dodged

        "Note the use of the work 'logged'. The bug has been around for decades. Do your logs go back as far the server build time?"

        Mine do (about 8 years). The first attempts to exploit this hole started coming in on Thursday afternoon.

  4. itzman
    Happy

    ALL of bashes features are obscure and rarely used.

    AFAICT ;-)

  5. Alistair
    Windows

    Bash patches and the flack.

    I've been following the discussions that this group of folks are having, on the mailing list. Quite closely.

    In our case (approximately 1600 linux hosts, all RHEL, covering 4->6 in numerous set ups, we've had 0 issues with the patches, we waited until the second fix came out before we pushed, and cfengine did the job it was built for so there was no overtime involved. The patch does not require a reboot, but in most cases we did an apache (or other) graceful restart, just to play it safe.

    Long and short - the issue was discovered, and about 4 days later a set of suggested fixes were discussed, and the first set of fixes was put in place, those were tested and one tester found an additional unique path that had a similar flaw. That opened up a new set of discussions because it was a substantially different path through the code. The debate on how to handle that took much longer since it required much more stringent logic to avoid breaking things in current use. If you went and read the list to see what they were discussing, you'd find that the secondary fix is in no way easy, and despite us already having a fix, will result in much larger code changes in the future.

    What scares the crap out of me on this front is all those landfill wifi routers out there that use cgi - many of them DONT use bash, but I'll bet that there will be a crapton that do and have lousy coding practices, and will NOT be updated anytime soon. Worm time anyone? *that* will be a pain in the tuchus when it comes down.

    These guys are bright, solid coders, applying solid common sense and strong skills to fix the problem, without breaking things that work the way they should. Not an easy job folks.

    1. Alan Brown Silver badge

      Re: Bash patches and the flack.

      "Long and short - the issue was discovered, and about 4 days later a set of suggested fixes were discussed, and the first set of fixes was put in place, those were tested and one tester found an additional unique path that had a similar flaw."

      It's worth noting that similar holes have been picked up in sh and ksh (at least), as people start poking round to see if other shells have problems.

      The problem with "legacy code" is that because it's legacy everyone assumes that it's safe to run and audited (this is a constant refrain raised here against updating things). The last few days (and the X bug last year) is a clear example of why such a stance is a logical fallacy and that ALL legacy code should be considered dangerous unless there's a recent auditing statement for it.

      Note that in all cases, holes in legacy software have generally been discovered within minutes of people going looking for them. The implication is that the vast majority of such code is a minefield of undiscovered issues.

  6. Barracoder

    Thank you...

    ...to the Phantom Downvoter, for amusing me with your single, stark downvotes on anyone who complains about downvotes.

  7. MarkSitkowski

    Bash bug? What Bug?

    I tried all the tests and potential abuses on bash, as distributed with Sun Solaris, and found that it doesn't suffer from any of the GNU bugs.

    Simple solution: install Solaris - I believe it runs rather well on x86 as well as SPARC.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like