back to article LibreSSL crypto library leaps from OpenBSD to Linux, OS X, more

The OpenBSD project has released the first portable version of LibreSSL, the team's OpenSSL fork – meaning it can be built for operating systems other than OpenBSD. The LibreSSL project, which aims to clean up the buggy and inscrutable OpenSSL code, was founded about two months ago by a group of OpenBSD developers, so it only …

  1. Anonymous Coward
    Anonymous Coward

    Two thumbs up to Theo DeRaadt ...

    ... and his Crew for having cleaned up the incomprehensible hairball of code that is OpenSSL.

    1. Voland's right hand Silver badge

      Re: Two thumbs up to Theo DeRaadt ...

      This pretty much terminates the discussion on the future of OpenSSL.

      Now watch how the linux distros will pick up this instead of any of the competing projects (even Linux foundation driven ones) the same way they have picked up OpenSSH, OpenBSD inetd, tftpd, etc.

    2. Anonymous Coward
      Anonymous Coward

      Re: Two thumbs up to Theo DeRaadt ...

      Not so fast.

      OpenSSL is indeed a complex hairbal of a problem, and I would thus be very suspicious that any team can get to grips with how it works in that time, let along replicate its functionality without introducing new problems along the way. Refactoring code is an art form in itself.

      So thumbs up for the effort, but there is a need for an independent review effort - if it withstands that, the team will have done a remarkable job.

      1. Anonymous Coward
        Anonymous Coward

        Re: Two thumbs up to Theo DeRaadt ...

        I agree that LibreSSL should be audited by as many people/institutions as possible, but please remember that many of the OpenBSD team have been doing this or very similar work for many years and they continue to do so with OpenBSD.

        There is possibly no single group better positioned to make the necessary code changes, outside of government acronym agencies where the funding is much better.

      2. Voland's right hand Silver badge

        Re: Two thumbs up to Theo DeRaadt ...

        Not so fast.

        I am not. Theo's lot has refactored for security and maintainabilty:

        1. Most of the BSD codebase (both Net and Free have taken back quite a bit from the Open tree).

        2. Most of the original unix utilities - the whole net tools/inet lot, cron, etc.

        3. Bind (talking of hairballs nothing can compete with Paul Vixie code - even openssl)

        4. God knows what else. 1,2,3 are just off the top of my head.

        If any team is capable of refactoring SSL it is them. Not anyone else.

        1. asdf

          Re: Two thumbs up to Theo DeRaadt ...

          >If any team is capable of refactoring SSL it is them. Not anyone else.

          I love their whole attitude of ok we can't trust the openssl devs as they are not responsible devs (ignore bug reports, all but new feature patches for years, etc) so we are going to bring it in house to our ecosystem and do it ourselves and if anyone wants to port our efforts here you go. It will rub many the wrong way like Theo often does but his baby OS (OpenBSD) is really in many ways the perfect OS to initially target. As you say few if any teams are more security conscious from the ground up and if anything they are kicking themselves for not having done this sooner due to their in house knowledge of crypto and security and the general sorry state of OpenSSL.

      3. Charlie Clark Silver badge

        Re: Two thumbs up to Theo DeRaadt ...

        So thumbs up for the effort, but there is a need for an independent review effort…

        That is a loaded statement. Code review is always good and should be part of the development process. However, let's think about the suggestion in the context of the OpenSSL debacle:

        1. The fork was started after a code review

        2. Any good fork should aim to pass at least all existing unit tests

        3. There already exists sophisticated penetration testing infrastructure for testing the known weaknesses of OpenSSL and discovering new ones both in it and LibreSSL

        4. Code counts - the best way to discover defects is to make the code available

        If LibreSSL can pass the existing tests then it is as secure as OpenSSL. Cutting a release will encourage the security experts to scrutinise it and competition here between the two projects can only be good.

        1. Michael Wojcik Silver badge

          Re: Two thumbs up to Theo DeRaadt ...

          If LibreSSL can pass the existing tests then it is as secure as OpenSSL.

          Utter rot. There is no suite of "existing tests" that fully vets the "security" of OpenSSL, and no such suite of tests is possible. And the phrase "as secure as" is essentially meaningless.

    3. Michael Wojcik Silver badge

      Re: Two thumbs up to Theo DeRaadt ...

      having cleaned up the incomprehensible hairball of code that is OpenSSL

      Evidence that LibreSSL is "clean"1 would be appreciated.

      While I too have much respect and some affection for de Raadt - a curmudgeon's curmudgeon if ever there were one - this looks to me like an extremely premature evaluation. Eliminating some of the architectural infelicities and archaic-platform kluges is useful2, but - and I've said this before too - the underlying problem is the maintenance-hostile coding style, and I don't see any sign that the LibreSSL team is interested in fixing that.

      1Personally, as I've noted before, I'm of the opinion that any code using KNF can't really be considered "clean". But that's just a quibble.

      2Though my bet is they'll take the platform cleanup too far. There are people using OpenSSL on many platforms that the LibreSSL team are likely to scorn. I suspect they don't care, and neither will their fan base, but it's a bit disingenuous to pretend that no one needs any of the stuff they're ripping out.

  2. asdf

    Corporations (like Google) need to step up.

    >says it has yet to receive a stable commitment of funding

    Well its not stable but I sent them $20 just because its that important. Reading their take on the OpenSSL spaghetti mess is also beyond hilarious. As a mostly C++ 11 coder its also gotten me interested in best practices in old school C (yes yes I know C++ is almost a superset of C but boy are they two different worlds even in mindset) so perhaps can help out eventually as well.

    1. Christian Berger

      Re: Corporations (like Google) need to step up.

      BTW if you are interested in software design in C, read "The Art of UNIX Programming". It's a completely different mindset to the C++ one.

      1. Destroy All Monsters Silver badge

        Re: Corporations (like Google) need to step up.

        Downvotes? What terrorist dislikes the "Art of UNIX Programming"?

        1. Cipher

          Re: Corporations (like Google) need to step up.

          Destroy All Monsters wrote:

          "Downvotes? What terrorist dislikes the "Art of UNIX Programming"?

          Clearly a splinter group of those terrorists reading The Linux Journal...

          1. This post has been deleted by its author

        2. Christian Berger

          Re: Corporations (like Google) need to step up.

          "Downvotes? What terrorist dislikes the "Art of UNIX Programming"?"

          In my experience there is a violent branch of the C++/Java/C# fans that completely hates that book. Unfortunately some of them are now found in what is called the "Freedesktop" movement.

    2. Anonymous Coward
      Anonymous Coward

      Re: Corporations (like Google) need to step up.

      "As a mostly C++ 11 coder its also gotten me interested in best practices in old school C (yes yes I know C++ is almost a superset of C but boy are they two different worlds even in mindset) so perhaps can help out eventually as well."

      I do hope you're not suggesting that if rewritten in C++ it would somehow be more readable because as someone who also is a professional C++ coder I can assure you it would not be. For something as low level as SSL you need entirely *explicit* code, not the implicit untraceable to the naked eye mess most C++ programs end up as with cascades of constructors and destructors constantly firing off from stack objects coming in and going out of scope and no one being 100% sure exactly what is going on especially if you chuck in the STL with all its behind the scenes memory allocation games. And what would you do with 2011 here? Throw in some lambda functions or auto types or maybe default template values just to confuse the hell out of anyone tracing the code in a normal editor?

      C++ has its place , but this sort of low level almost to-the-metal code is not it.

      1. Anonymous Dutch Coward

        Re: Corporations (like Google) need to step up.

        @boltar "I do hope you're not suggesting"... well I don't see him suggesting that, do you? Don't let that stop your rant though... very enjoyable ;)

        1. Anonymous Coward
          Anonymous Coward

          Re: Corporations (like Google) need to step up.

          "well I don't see him suggesting that, do you?"

          Seemed like the implication was there. If not then doesn't matter, the point stands on its own.

          "Don't let that stop your rant though... very enjoyable ;)"

          I'm curious - can you just explain why people like yourself class any opinion you disagree with as a "rant" or denigrate it in some other way? Its a very adolescent thing to do.

      2. asdf

        Re: Corporations (like Google) need to step up.

        >I do hope you're not suggesting that if rewritten in C++

        I was actually suggesting the opposite. I personally want to be more versed in best practices modern C coding (and comfortably read the ugly old stuff). Refactoring OpenSSL in C is herculean task alone, rewriting it in C++ would take a very large corporate or government effort and as you rant wouldn't necessarily be an improvement even if done right. A lot of OpenSSL problem is their crappy ass APIs which due to compatibility would be a problem regardless of the toolset/languages used.

      3. Ken Hagan Gold badge

        Re: "C++ has its place , but this sort of low level almost to-the-metal code is not it."

        From your heartfelt complaints, I infer that you were once exposed to some complete idiots who took the C++ language spec as a challenge, and you've developed a hyper-sensitivity to feature abuse as a result.

        For code like this, I'd reckon that idiomatic C++ would differ from idiomatic C only in using constructors and destructors to automate memory management and structure initialisation/cleanup. There might be a large-integer class with overloaded arithmetic operators, but if you can't handle using infix operator notation for integer arithmetic then you probably can't handle the theory behind SSL.

        I'd expect an almost line-for-line correspondence between the two code bases. I'd expect the two compilers to generate almost identical code. I'd expect an experienced C coder with only a passing knowledge of C++ to be able to read and maintain the C++ safely.

        C++ was largely developed by experienced C coders who wanted to make it easier for themselves to write C code, and one of the basic design principles is "no room for a lower level language, except assembler", so all the bare-metal tricks beloved by C coders are valid C++. A Real Programmer, of course, can write FORTRAN 66 in either language.

        1. Anonymous Coward
          Anonymous Coward

          Re: "C++ has its place , but this sort of low level almost to-the-metal code is not it."

          "From your heartfelt complaints, I infer that you were once exposed to some complete idiots who took the C++ language spec as a challenge,"

          Pretty much in every company I've worked in, usually by people who've read a design patterns book and seem to think they need to apply as many patterns as possible and abstract as much as possible with whatever problem they're given regardless of whether it warrents it or not. Usually it turns out these sorts are nothing more than lego brick coders who have no real feel for coding but they can bluff their way along by virtually cutting and pasting methodologies into their code.

          "C++ was largely developed by experienced C coders "

          Don't get me wrong - I like C++ , its a far better OO version of C than that Objective-C abortion Apple seems to love. I just don't happen to like the way it tends to be used. The problem is that with C if you don't know what you're doing you won't get very far - your program will more than likely crash pretty fast and often. But with C++ if its kept at a high abstract level with liberal use of the STL and other libraries lot a lot of people can just about manage to get something working.

        2. Anonymous Coward
          Anonymous Coward

          Re: "C++ has its place , but this sort of low level almost to-the-metal code is not it."

          I'd expect an experienced C coder with only a passing knowledge of C++ to be able to read and maintain the C++ safely.

          Steady on...

          1. asdf

            Re: "C++ has its place , but this sort of low level almost to-the-metal code is not it."

            >I'd expect an experienced C coder with only a passing knowledge of C++ to be able to read and maintain the C++ safely.

            Depends. If you get someone that loves template meta programming and takes it to the extreme then even an experienced C++ developer might have problems maintaining the code. Seen this before.

    3. Dan 55 Silver badge

      Re: Corporations (like Google) need to step up.

      They already stepped up, they wrote blank cheques to the OpenSSL group (now part of the Linux Foundation), who, in the same time as the OpenBSD time, have sent out an e-mail with some vague objectives about what they're going to do.

      It'd have been quicker for them just to set fire to that money.

  3. Anonymous Coward
    Anonymous Coward

    "Note that this is still considerably fewer platforms that the original OpenSSL library supported. But OpenSSL's portability approach had become one of extreme overkill, with the code incorporating numerous hacks and workarounds to make it run on such outdated platforms as VMS, OS/2, NetWare, 16-bit Windows, and even DOS."

    While LibreSSL doesn't offer the plethora of OS support that OpenSSL does on paper, even the OpenSSL devs have admitted this and was referenced by an article on The Register from the same author:

    "What's more, the development team admits that it has had no strategy for maintaining the code base across the many platforms to which it has been ported. Some are legacy platforms and the developers currently have no access to them, so there's no telling whether today's code still runs on them."

    Do we really need OpenSSL (or LibreSSL) supported on platforms that make it harder to support the platforms of today in a secure and software auditable way?

    1. Roo
      Windows

      "Do we really need OpenSSL (or LibreSSL) supported on platforms that make it harder to support the platforms of today in a secure and software auditable way?"

      For the cases where the answer is "yes", the tidying up effort should make forking & porting easier for the folks who need to support an oddball, so all in all I think it's a win.

    2. Anonymous Coward
      Anonymous Coward

      "Do we really need OpenSSL (or LibreSSL) supported on platforms that make it harder to support the platforms of today in a secure and software auditable way?"

      Do you know where those platforms are still in use, don't you? And no, it's not museums...

      "Some are legacy platforms and the developers currently have no access to them,"

      It looks OpenSSL developers never used a virtual machine?

      1. Anonymous Coward
        Anonymous Coward

        Do you know where those platforms are still in use, don't you? And no, it's not museums...

        Then the either the companies that use them or the companies paid to support those old OS's can then take OpenSSL or LibReSSL and port it to those platforms.

        When the companies that wrote those OS's have considered them deprecated, then continued development and support of them is the responsible to other parties.

        "It looks OpenSSL developers never used a virtual machine?"

        Like the Alpha processor, the Motorla 68000, various PowerPC, MIPS and old ARM's are not easily VM'd.

      2. Michael Wojcik Silver badge

        "Some are legacy platforms and the developers currently have no access to them,"

        It looks OpenSSL developers never used a virtual machine?

        Which virtual supervisors or hypervisors support OS/2 Warp?

        I know one that supports zOS, but it ain't cheap.

        And considering that, prior to Heartbleed, the OpenSSL development team was a couple of people, they haven't really had the resources to test every release across every supported platform.

        Perhaps a bit of education before casting aspersions would be useful.

  4. Mark Solaris

    As a long time code porter, I want to punch in the face anyone that uses quirky C++ methods in their code, it's a PITA to backport it. The KISS principle should be ingrained into people before they get a keyboard.

    1. Anonymous Coward
      Anonymous Coward

      "The KISS principle should be ingrained into people before they get a keyboard."

      Tell that to the C++ standards commitee.

      1. Anonymous Coward
        Anonymous Coward

        If anything C++ has been simplified a great deal in the last and forthcoming iteration. It's the lack of deprecation that makes it seem otherwise. I'm at a loss as to why they don't deprecate features that are superseded by something simpler/better.

        1. Ken Hagan Gold badge

          Deprecation achieves very little unless you can persuade people to re-write old code. Otherwise, compiler vendors have to provide two "modes" of compilation: strict and legacy.

          Much the same goes for loud compiler warnings. People just compile their "old" code with the warnings off. However, these *can* be used to ensure that nasty old practices are not accidentally re-introduced in a modern codebase.

          1. Anonymous Coward
            Anonymous Coward

            Deprecation achieves very little unless you can persuade people to re-write old code.

            I wasn't really suggesting that people be forced to rewrite old code.

            Otherwise, compiler vendors have to provide two "modes" of compilation: strict and legacy.

            I couldn't say for other compilers but G++ at least lets you set which version of the standard you wish to compile with.

            Much the same goes for loud compiler warnings. People just compile their "old" code with the warnings off. However, these *can* be used to ensure that nasty old practices are not accidentally re-introduced in a modern codebase.

            If the standards committee deprecated more stuff we'd at least get warnings when compiling C++11 code that contained superseded C++98 features. As it is, setting the -std flag on the compiler only prevents you from using features introduced after that standard's publication.

    2. Zama

      Would you mind shedding some light on what "quirky c++ methods" are and why they're so hard to port?

      1. Anonymous Coward
        Anonymous Coward

        One example - Multiple inheritance in an interface class X which derives from A,B which defines a pure virtual which is overridden by a pure virtual covariant return method in a class Y that derives from X.

  5. Will Godfrey Silver badge
    Thumb Up

    Good work

    I'm very pleased to see this, if for no other reason than it puts a rocket under the original devs.

    1. Michael Wojcik Silver badge

      Re: Good work

      The OpenSSL team was well aware of the problems with their code base long before this. The issue was one of resources, and by far most of their income came from chasing particular revenue streams (like FIPS certification). It was economically infeasible for them to do the kind of work the LibreSSL team has done, and no one else stepped up to do it, because using OpenSSL as-is was cheaper.

      Now, post-Heartbleed, the OpenSSL team is much larger and better funded, and they are in a position to address technical debt.

      Frankly, I don't see the existence of LibreSSL as much of an additional incentive. OpenSSL continues to have the name recognition. It has the installed base. It has FIPS certification, which is critical for sales to the US government - a not-insignificant market for many of the ISVs using OpenSSL. If I were on the OpenSSL team (rather than just one of the many people who use it, monitor the mailing lists, and so on), I wouldn't give a rat's ass about LibreSSL, except to check their commits periodically to see if there were fixes that applied to the OpenSSL trunk as well.

  6. Anonymous Coward
    Anonymous Coward

    Code size

    I had often assumed that the size of the OpenSSL code was enormous - many of these articles leave you feeling that they are dealing with millions of lines of code.

    But actually the amount of code is not much. I think a lot more people would contribute if they knew this.

    1. Bronek Kozicki

      Re: Code size

      No, they wouldn't. Code size is not the only thing that matters - coupling is the same, if not more important. Think of it as the possible number of permutations in a group of elements (elements being design artifacts, ie. functions in C language). In a design with qualities of a hairball (anything connects to anything), the number of possible permutations can be huge, despite the total number of lines "merely" going into many thousands (below million). In order to understand it, you need to read it all and then build mental model of everything there is. That makes for very high barrier to entry.

      The purpose of good software design (each language provides own design tools for this, in case of C that would be private headers, static functions etc.) is to control and lower the number of possible connections, thus lowering the overall complexity and the cost of reading and contributing code - despite total code size remaining roughly the same, or perhaps even slightly larger (depending on design tools used).

      Of course, LibreSSL didn't set to increase the codebase with design artifacts. They set to remove all dead code first, which obviously is a very good way to start such a project. They are also limited by public API of OpenSSL which makes lots of private functionality available to users unnecessarily. But they are to a good start and I wish them well, enough to setup monthly donation.

  7. btrower

    Code is truly awful, but sadly not unusual

    I took a look at the source here: http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/

    Code like this abounds and it seems to me especially in security code (the last place it should exist). The very first file I opened, ssl_asn1.c, contains multiple points of exit and goto statements. It is a festival of mutant coding practices. I am looking right now at the classic evil code comment:

    /* can't happen */

    I am not kidding. My eye came to rest there because the line is indulging in the journeyman C programming stylistic *error* of failing to put braces after an if statement because the programmer believes with all their heart that they are prescient enough to foresee that nobody will ever mistakenly add another statement and forget to put in the braces. This, of course, is ruled by Murphy's law and does, despite the programmer's confidence it is 'impossible', in fact happen. Sigh.

    Code that contains unconditional jumps like goto statements, unstructured breaks, multiple points of exit, effective multiple points of entry, etc is much harder to debug than it needs to be. One of the first things I would do to fix stuff like that would be to restructure and correct the many stylistic problems. Code like that invariably hides a variety of bugs. That is especially true of old code that has been visited by multiple programmers because the code is only as strong as the weakest programmer that touched it. Cleaner code that follows good practices gets touched less.

    Spot checking to make sure that I am not looking at one aberrant file, the following are similarly impaired:

    ssl_lib.c, s3_pkt.c, t1_enc.c, pqueue.c, smime.c, openssl.c, rsa.c, ec.c, x509_req.c

    It is a gruesome body of code. Whoever said it was not large cannot possibly have much experience with this type of thing. I count more than a quarter million lines of code. There is enough that I would be inclined to actually write tools to do a lot of the boilerplate cleanup.

    Adding more programmers is not necessarily productive, so even with an unlimited budget your best bet is to go with a finite number of programmers. However, off the top of my head, where code quality trumped budget and the budget was effectively unlimited I would be inclined to assign a fairly large team to fixing this on the order of three dozen people. Assuming a target blended hourly rate of about $80.00/hr and adjusting for the inevitable overruns, a project like this would cost something approaching $10 million if done through a large consulting firm. I have worked on projects with much larger budgets ($250 million on a couple), but this would still be a large project.

    Unless a very, very clever small team of programmers builds tools to fix the code, or it is funded as above by big players, or a canny coordinator can crowd-source enough hands, this code is not going to be very trustworthy. I am not sure if it would be better to start from scratch.

    The above assumes that openssl can actually accomplish its ultimate purpose and I personally am doubtful. I think our entire security infrastructure is similarly impaired in implementation, design, architecture and ultimately philosophy. It was designed in a much more naive time. Security has now escalated into a profoundly adversarial situation involving very well funded organized crime, states and powerful industry players. Many of the assumptions underpinning current security thinking are patently false.

    The most poisonous aspect affecting SSL is the demonstrably false assumption that the current chain of trust is trustworthy. That includes more than just the CAs. The money to fix this properly is definitely there and the people controlling those funds are most certainly aware that the system is broken. It remains broken because an insecure network for most of us with its attendant increasingly open cyber warfare serves their purposes more than a genuinely secure network.

    Maintaining code like this is a heroic effort. Both teams engaged in this deserve credit for taking on a very difficult and thankless task. Even though mistakes abound, the code has a heritage reaching back to times when some of their practices were considered benign or even 'best practice'. Having reviewed it, I am arguably one of the people responsible in a way and unless I get some sort of divine inspiration, I will not be correcting the many issues with this.

    Technically, I think that it needs to be mentioned that many practices such as running regression test suites are necessary, but not nearly sufficient. They are *more* necessary when code looks like the openSSL stuff, but they are proportionally less sufficient.

    I have been designing and reviewing security code for decades now. It is *very* difficult. In fact, I am less confident now than I was twenty years ago. Even if the openSSL code was carefully rebuilt by seasoned programmers I would not trust significant financial transactions to it, let alone military secrets.

    1. Anonymous Coward
      Anonymous Coward

      Re: Code is truly awful, but sadly not unusual

      Have an upvote for sharing your thoughts but...

      > However, off the top of my head, where code quality trumped budget and the budget was effectively unlimited I would be inclined to assign a fairly large team to fixing this on the order of three dozen people. Assuming a target blended hourly rate of about $80.00/hr and adjusting for the inevitable overruns, a project like this would cost something approaching $10 million if done through a large consulting firm.

      ... I'm not surprised. If projects done through large consulting firms are what you usually experience, the level of single-programmer productivity is such that you'd have to size it in that manner and expect - as you do - inevitable overruns.

      I have a strong feeling that a single one of the security-conscious and -experienced OpenBSD-affiliated programmers working on libressl who do this because it's a passion rather than a contractual obligation run rings around any whole three-dozen team coming from IBM/Accenture/WiPro/etc or any combination of them.

      And while that three-dozen team might dot all the i's and cross all the t's in terms of abiding by your favourite good/secure coding standard, I'd still be more likely to trust whatever the OpenBSD team ends up using themselves.

      > Even if the openSSL code was carefully rebuilt by seasoned programmers I would not trust significant financial transactions to it, let alone military secrets.

      They have put this out in record time and I don't think anyone suggests they haven't made significant improvements over what OpenSSL is. Your team would likely still be at the PowerPoint stage phrasing vision and mission statements and various sociopaths would scheme to get themselves into politically advisable positions.

      Seeing that people have to use something right now and over the next year, what do you suggest they use instead?

      1. asdf
        Thumb Up

        Re: Code is truly awful, but sadly not unusual

        >Have an upvote for sharing your thoughts but...

        Wow an AC comment that hit it out of the park. Don't see that very often.

      2. btrower

        Re: Code is truly awful, but sadly not unusual

        My experience is not at all limited to megaprojects. I have worked on pretty much every size and type of software. My experience is a bit limited because I have almost always been on the development side rather than maintenance. I think that programmer productivity does fall with project size, but not entirely as much as you seem to think, nor for the same reasons. You cannot take a few exemplary and successful small projects and their developers and use that as a yardstick for even other small projects, let alone large ones.

        Take a look at the actual projects and source code on sourceforge; count code age and lines. Does it exceed daily productivity by paid professionals from large consulting firms? I have seen a lot of code from both sources and I would say that it does not. Most big projects have been less cost-effective than I would have liked, but not all. I worked on a project costing tens of millions of dollars with Sybase consulting and they kicked ass. I also worked on a huge multi-billion dollar production system for one of the largest firms in the U.S. and it is the only non-trivial project where we managed to come in with quality code both ahead of time and below budget.

        Re: "size it in that manner and expect - as you do - inevitable overruns."

        That was a thumbnail quote based on a line count and inspection of a sampling of code. It may seem strange to some, but in my experience a survey of code quality and line count has been the best indicator of the time required. I expect that if I had that budget and was allowed to form my own team that I could come in below that budget. However, overruns are inevitable on most projects because they would not have been started at all if more realistic estimates were made. Writing software, even with an existing system as a spec (huge leg up), is still labor intensive and difficult. I have been a career consultant and worked on a lot of different sizes and types of projects including in software companies as such like Sybase and Microsoft. My experience, as far as I can tell, is about exactly in line with what generally happens.

        BTW -- I have worked on large projects with both IBM and Accenture and if you think you are better than their top guys I suspect you have not met their top guys. They *do* pitch out some less than stellar programmers on huge projects, but they attract some really good people. I was out a bunch of times with IBM, Andersen and then Accenture and none of our projects failed to deliver. You could have done it cheaper, but when you are spending $250 million dollars on a mission critical system that has to be there on time, it is not prudent to wing it.

        I am near certain that I could beat the average large consulting firm project in terms of time and budget. I am also near certain that I would not be able to beat them if they fielded their best people. In no case could I possibly offer the guarantees that they can. One of my Andersen projects came in on time because there was a series of something like $5 million penalties for coming in late against milestones and they had deep enough resources to insure they made it.

        Whether we like it or not, cost and time over-runs are a fact of life in complex projects. Failing to plan for that is planning to fail because of that.

        Re: One BSD developer beating thirty developers from a large consulting firm.

        In my experience you would lose that bet in a spectacular way. The average team member would probably be a bit worse, but the kind of consulting dollars available on these projects attracts some very good people. Chances are the same guy would be offered $200 per hour to work on the project with the big team as well as being offered much greater resources and helpers to amplify his productivity. If he is really serious about his craft, he will join the big team. Heck, he might even learn something.

        Career programmers with thousands of hours of hands-on experience and credentials like Master's degrees in Engineering and PhD's in math are not guaranteed to be the best software developers. Some are even bad, but in my experience they are pretty good.

        Whatever you can say about the openSSL code quality, I would say that it reflects work below the median quality of most working professional programmers. That is not to say that the programmers who worked on it are less than mediocre. It is an enormous amount of intrinsically difficult code in a language that many programmers find challenging. My hacks may not include gotos or multiple points of exit or mutant stylistic habits, but they can still be pretty ugly. If you only have time to do the hack, even the best programmer will produce poor code. Is there an experienced working professional programmer anywhere that has never been boxed in like that? I highly doubt it.

        Re: "I'd still be more likely to trust whatever the OpenBSD team ends up using themselves."

        This could go either way, but I am pretty sure that, given the budget I mention I could do better than they will end up doing and so would Accenture or IBM.

        Re: "Your team would likely still be at the PowerPoint stage phrasing vision and mission statements"

        This might have been true in the past, but I doubt it is now. Given the budget I mention, a couple of people on the team would be formally interfacing with management and users. It is possible that the team might *also* produce management friendly documentation. In the past I have worked on ISO 9000 projects where various documents (a lot of them) were a formal requirement. You might want to have a bit of polish on the presentations and documentation too if you were spending tens of millions of dollars on a project.

        Re: "and various sociopaths would scheme to get themselves into politically advisable positions."

        I agree with you here. The climbers are everywhere and they muck up the works with their schemes. Since they are devoted to climbing and you are devoted to your actual job, they end up winning a lot even though they are visible to at least some. This may be over-represented in fat and happy environments, but no organization is immune to this, including open source.

        Re: "Seeing that people have to use something right now and over the next year, what do you suggest they use instead?"

        There are good arguments in favor of both the existing project and the forked one. I expect that the SSL code generally will benefit by this situation, though it may be painful to the openSSL maintainers. Either should do, but if I had to make one choice I would choose the original over the forked version.

        SSL is important enough that it should support a budget in the tens of millions to seal it up. However, even if SSL achieves perfection with respect to its design and code, it will still suffer from the many architectural and philosophical difficulties the whole system has.

        Hearbleed was about the worst security bug I have ever seen in terms of both breadth and depth. It potentially affected every net connected device in the world, including ones that did not even use openSSL. I am not making light of it, but if you think about it, the only thing that has changed before and after is that we found out about this specific misfeature and we fixed it. We are not in good shape, but we are in better shape than before.

        My estimation of our security situation has not changed. If you look at prior postings of mine you will see that security and privacy are big issues for me and I have been pretty blunt that security is hopelessly broken. That has not changed -- up or down.

        You have to ask yourself why IBM, Red Hat, HP, Apple, Microsoft and other large players have not put together the necessary budget and had this done. Even at a commitment of $20 million it is a drop in the bucket for them. The weakness of the current openSSL code base negatively affects everyone, including people who do not even use it. They all have people who would have, like I did, look at heartbleed and know that even by our shabby standards of security it had a very serious problem. Given a look at the code base I would say that it is almost certainly still broken.

        Perhaps they know that the systemic problems render an SSL fix pointless. Maybe they are under orders from some government agency not to rock the boat. Maybe they really don't think the openSSL code is that bad. I can't think of a scenario where all the big players look good here.

        1. Anonymous Coward
          Anonymous Coward

          Re: Code is truly awful, but sadly not unusual

          Oh well, have another upvote for sharing your opinion.

          We obviously have very different views on large consulting projects and I bet it's on both sides nothing but anecdotal evidence mixed with our individual levels of sarcasm.

          You find that making curly braces around "if" code bodies improves readability/maintainability/security. In fact, I do, too. But it's really a tiny thing in the grand scheme of things, don't you think? Code tests with coverage reports I think we can agree is a whole lot more important and when done extensively, will tell you when you added a second line to an if body incorrectly, too.

          From what I've heard, OpenSSL does not have anywhere near enough tests and they are a pest to write. I confess I don't know what the plans of the libressl team are with respect to this.

          Actually, writing this makes me want to see if I can contribute tests. Presumably my C skill is far too rusty but perhaps...

          PS:

          Re Apple, I thought they have their own SSL implementation, with its own set of security vulnerabilities and questionable coding style? http://www.theregister.co.uk/2014/02/23/goto_fail_a_symptom_of_softwares_cultural_malaise/

    2. Ken Hagan Gold badge

      Re: Code is truly awful, but sadly not unusual

      /* can't happen */ ??

      Isn't that most portably spelled "abort()"?

      If the compiler can prove your assertion, it will generate no code. If it cannot, then it will cost you a few bytes of code. Either way, each time you change the surrounding code, the compiler will re-check. On any given platform, there may be non-portable alternatives that turn mistakes into a compile-time error.

      1. This post has been deleted by its author

    3. h4rm0ny

      Re: Code is truly awful, but sadly not unusual

      >>"Even if the openSSL code was carefully rebuilt by seasoned programmers I would not trust significant financial transactions to it, let alone military secrets."

      What about BoringSSL, though? Google has *a lot* of resources and they're doing their own fork of OpenSSL. They say they want to co-operate with OpenSSL and LibreSSL as well, so that's a lot of potential development resource we're talking about.

      I've seen a few disasters where instead of improving what was already there, people tried to clean slate everything. It's very tempting to do that when you're a developer but look at Netscape / Mozilla. They lost massive ground to IE because they decided to re-write, rather than re-factor. And it took a long time to catch up.

      As someone else said, what's the alternative? I found your post very interesting btw, so this isn't my arguing with you. I'm just querying if you genuinely believe a whole new project would actually be better.

      1. Anonymous Coward
        Anonymous Coward

        Re: Code is truly awful, but sadly not unusual

        Google already have their OpenSSL alternative in place and running. It's a Play Services deployed (and serviced) SSL layers.

        Developers can use the core OpenSSL implementation that comes with Android, or the Google Play serviced one.

        http://developer.android.com/reference/com/google/android/gms/security/ProviderInstaller.html

      2. Michael Wojcik Silver badge

        Re: Code is truly awful, but sadly not unusual

        It's very tempting to do that when you're a developer but look at Netscape / Mozilla. They lost massive ground to IE because they decided to re-write, rather than re-factor.

        Eh? The Mozilla code base was derived directly from Mosaic. It and the IE (formerly Spyglass) code base share a common ancestor in Mosaic, but in no sense was Netscape a "re-write" of anything in IE.

        That aside, there are other SSL/TLS implementations that are independent of OpenSSL, of course. Several of them - GnuTLS and Apple's implementation are the most notorious - have had some of their own shortcomings exposed in recent months. Neither independent projects nor attempts to clean up the OpenSSL code look like the royal road to a better implementation.

        If I had the mandate and adequate resources, I'd start by rewriting all of the OpenSSL code for readability. Then pull out the architectural mistakes (like the rightly-maligned suballocator) and refactor where reasonable to reduce redundancy. As far as I know, no one's taking that route; certainly the LibreSSL project hasn't.

    4. Ommerson

      Re: Code is truly awful, but sadly not unusual

      While we're at it, a cursory glance at ssl_asn1.c shows plenty of pointer de-referencing too, without so much as a check for NULL or an assert().

      Quality stuff that wouldn't get past my code review if one of my team developed it.

  8. PyLETS

    conflicting objectives

    That's probably why the code got into a mess. Having a good testsuite will certainly help in refactoring. Problems this library has to deal with include doing integer arithmetic securely on 4096 bit numbers and larger and at high performance on different CPUs. And you can't afford to leave any clues in memory which might be reallocated to a different process afterwards. So you've got all this non-standard stuff done in many different ways, and need to avoid integer overflow and stack and heap smashing bugs as well, none of which you can develop automated tests for until you know about them.

    1. Ken Hagan Gold badge

      Re: conflicting objectives

      "And you can't afford to leave any clues in memory which might be reallocated to a different process afterwards."

      Perhaps I'm just playing Devil's Advocate here, but if you are running on an "OS" (and I use the term loosely) that doesn't zero pages before handing them to another process, then you're wasting your time worrying about security.

      1. Anonymous Coward
        Anonymous Coward

        Re: conflicting objectives

        It's a bit more complex than that... you have to keep that memory "safe" as long as you have those data in memory. You shouldn't be anle to dump a process' ,memory and read the keys from it....

      2. Michael Wojcik Silver badge

        Re: conflicting objectives

        if you are running on an "OS" (and I use the term loosely) that doesn't zero pages before handing them to another process, then

        ... you're running on an OS which is not Orange Book C2 compliant - this is the "Object Reuse" requirement. I don't know of a contemporary virtual-memory OS which isn't at least C2.

  9. Anonymous Coward
    Anonymous Coward

    Trust + Compilers

    Maybe someone more knowledgeable than me can answer --

    What is the feasibility of introducing malware into the most common compilers?

    So, for example, when the compiler detects that it is compiling a certain type of code or a specific project (I realise this is hard), then it injects the backdoors at compile time.

    Surely this is the next level up in the hierarchy of where to 'break the internet' ?

    Do people check for this sort of attack, or regularly audit the source code of the compilers?

    Or audit decompiled code?

    1. h4rm0ny

      Re: Trust + Compilers

      >>"What is the feasibility of introducing malware into the most common compilers?"

      Theoretically possible but extremely difficult to do with Open Source. Everyone can see the commit history for Open Source projects such as GCC and privileges to commit tend to be a short list. So anything untoward introduced into the source code has an extremely high chance of being noticed and it's a short list of people who can do so. PHP source was briefly compromised back in 2010 when someone managed to fake a developer's credentials to make a commit. But they only added their name to the credits and it was still picked up and cleaned out. Again with PHP, they had a possible loss of developer credentials when their wiki server was compromised. They responded by doing a full review of all commits since the time the vulnerability was introduced and things were fine. So in short, getting your exploit into a public code base with a small number of approved and active developers is very hard and likely to be noticed quickly even if you do.

      So if you can't get your malware into the compiler's source code, you're left with trying to sneak it in as a binary that doesn't match up with the actual source code. This too would be very difficult. Many different groups compile their own binaries from source. They also compile regularly and frequently. If you download GCC for your Ubuntu distribution, someone hasn't been sitting there compiling that binary by hand. It's compiled automatically from automatically obtained source whenever changes happen and everything is signed with public keys. And multiple mirrors, too. Even if you sneaked your own tweaked binaries onto a server, it would be tricky to get it out there consistently and keep it there.

      So basically very hard to compromise the source, very hard to get binaries out there that don't match the source. That's one of the big advantages of Open Source - it's not that the code is inherently more secure, but that it's well protected against deliberate subversion by the providers.

      I welcome corrections on the above if I missed anything.

      1. Anonymous Coward
        Anonymous Coward

        Re: Trust + Compilers

        What about if someone managed to pwn the master repository for the source code and inject the hack such that they cover the fact the file was altered? Perhaps slip it into a little-used part of the code or split it into several pieces, each piece lying somewhere more plausible but when the whole thing comes together, they can all link together?

        1. h4rm0ny

          Re: Trust + Compilers

          >>"What about if someone managed to pwn the master repository for the source code and inject the hack such that they cover the fact the file was altered?"

          Okay, I kind of talked about the scenario but more detail would probably be useful. All of the Open Source code is covered by version control systems. Sometimes Subversion, often GIT. Whatever is used anyway, the principle is the same - you track the changes that are made by the developers so that you can review them, roll them back, work on seperate branches et al.

          Suppose you did have complete control over a server that was the main repository for the code, from which others normally pulled (took their copies). The others don't get their copies by just copying over files, they get the changes made to the version control system. So if you changed a file without the version control system noticing somehow, the others still wouldn't get it because they're just requesting a change history (with all the developer comments, commit times, etc.). Furthermore, the original version control system would normally pick up the changes you had made as local uncommitted modifications which would stand out like a sore thumb. In order to avoid that, you'd need to compromise the version control system. Which in our full access scenario you could do, but that still wouldn't help you get your exploit onto the other copies of the repository because they only update according to published changes. And once you publish changes we're back to the fact that you're no longer covering that the file was altered. It's very tricky and I'm not actually sure how you would pull something like this off.

          >>Perhaps slip it into a little-used part of the code or split it into several pieces, each piece lying somewhere more plausible but when the whole thing comes together, they can all link together?

          I get what you're saying - it's kind of the movie scenario where someone goes through security in an innocent looking wheelchair and then the arm becomes a gun barrel and the battery opens to reveal the handle-bit and it all clips together to make a weapon.

          But happily it doesn't translate into that in practice. At least not unless you're an absolute genius. For a start, the more areas of the code you alter, the more likely you actually are to attract attention. Development teams tend to split into different areas of the code that they handle. If you've got one area where someone is inattentive maybe you can compromise that. If you're treading on everybody's areas, someone will notice. Especially as you'll (presumably) be doing this under one account which would look odd, or multiple developer accounts, which would increase the risk of someone going: "I didn't write that!".

          Ditto really for the "quiet" area of the code. If I were a hypothetical criminal mastermind doing this, I'd actually bury my changes in an area where there were a huge number of commits and my change would hopefully be lost in traffic. I'd also choose a very busy time in the project where the commits were flying thick and fast.

          If you picked an area that hardly ever changed, you'd just get a lot of developers looking at their version control tools going "huh - why has that module suddenly changed".

          I hope all this doesn't come across as me shooting down your ideas. They're all very legitimate questions and exactly the sort of thing someone smart but not familiar with the process would ask. As I say: this is one of the great strengths of Open Source. (The other, imo, being surety of long-term code availability and potential to fork it yourself if needed).

    2. Anonymous Coward
      Anonymous Coward

      Re: Trust + Compilers

      After a bit of research I found a paper from way back in 1984 that talks about this technique.

      http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf

      1. Ken Hagan Gold badge

        Re: Trust + Compilers

        And one of the big differences between now (as well summarised in h4rm0ny's reply) and then is the off-the-cuff remark that Thompson was able to give the first version of his most evil compiler (that didn't need the hack to appear in the source code) to all the relevant people under the guise of an update. I don't believe anyone could do that now, so the hack would always be in plain sight if you went looking for it.

    3. btrower

      Re: Trust + Compilers

      @h4rm0ny is correct with respect to a specific contemporaneous attack vector. However, there are other attack vectors that could be exploited and we have proof that some attacks on our security infrastructure were launched long ago with industry collusion.

      I have the source code for the compiler I principally use (tcc) for small tools. Running my own version of the compiler does not give me much confidence it is not compromised. There are not all that many compilers about and if the NSA wanted to place evil code into C compilers, it would not be much harder to place the compromised code into all the binaries or for that matter in the operating system, BIOS, editors or hardware.

      You have to take the safety of many things on faith and a breach in any one leaves you open to at least some type of attack.

      The following says "hello world" in the visible source, but displays something different from a hidden stream in that same source file. If your IDE writes and only compiles an evil stream variant of your code, how would you know?

      echo hello, world>hello.source

      echo Goodbye cruel world.>hello.source:nsabackdoor

      echo compiled:

      more <hello.source

      echo running:

      more <hello.source:nsabackdoor

      The above would likely be quickly found by the community. However, MS could have, under a secret order from the NSA, put in an invisible alternate stream mechanism and OS code to make sure that an evil variant is always created, and silently presented. The OS can easily intercept and quiet any messages that hint at the evil code. It only takes a single compromised executable to mess things up and how many can say that they built from scratch, audited the assembly code for every binary and proven their hardware is secure?

      Unfortunately, the more you look at security the more you realize how fragile it is. It is fantastically difficult to mount a defense against a targeted attack from a well resourced and determined attacker. I have very little confidence that anything I use can withstand attack from all of the potential attackers out there.

      I could describe many specific weaknesses but there are so many it is hard to know where to start.

      As a practical matter, unless you are a programmer with a fair bit of knowledge about this area, you basically just have to trust other people to keep your systems safe. Even if you are a programmer with a fair bit of knowledge about this area, unless you have significant resources *and* a bit of luck you are not all that much safer.

      It is sad but true that you cannot really trust any of the involved devices even right down to the silicon. You should be able to, but years of neglect by white hats and enormous efforts by varieties of black hats (the worst being the ones who actually think they are white hats) has made every bit of the system suspect.

      A much more certain mitigation strategy is to work to shift liability to a party capable of taking it on. For instance, if you are going to do banking online, you follow all the rules of the institution providing the service and if your account is compromised, they were the custodians and have to fix it. If something needs to be kept safe, it is best to take it offline.

      1. This post has been deleted by its author

  10. Gordon Shumway

    Cleanup effort by OpenSSL

    ... so they have been unable to get their acts together for sixteen years, and yet there are some who think at the wave of a magic wand they suddenly become competent.

    Give me a break.

    "So, you are a PhD. Just don't touch anything."

  11. Anonymous Coward
    Anonymous Coward

    Pity they dropped Windows support

    Not because I like Windows, but because some of the larger cross platform applications that use OpenSSL probably won't be able to switch to this wholesale. :(

    1. Anonymous Coward
      Anonymous Coward

      Re: Pity they dropped Windows support

      But everything starting with "Libre" is today managed by a bunch of code-extremists whose only aim is to destroy Windows. I've seen already many "cross-platform" project trying not supporting Windows, because now "cross-platform" should just mean "support the n-thousand versions of something derived by Unix".

      It's also funny they like the "Libre" moniker so much, everything that used it in the real world turned into a bloody dictatorship..... but that tells a lot about the mindset of those behind that. They don't se code and software as a job or products, for them code *is* politics and a way to shape the world.

      I'll stay a way from such fanatics.

      1. h4rm0ny

        Re: Pity they dropped Windows support

        >>"But everything starting with "Libre" is today managed by a bunch of code-extremists whose only aim is to destroy Windows. I've seen already many "cross-platform" project trying not supporting Windows, because now "cross-platform" should just mean "support the n-thousand versions of something derived by Unix".

        I can't speak for what the developers are like as I don't know them, but I can install LibreOffice on Windows and have done in the past. Unless something has changed, I believe that to still be the case.

        >>"They don't se code and software as a job or products, for them code *is* politics and a way to shape the world."

        It isn't inherently bad to want to change the world or have a higher motivation for doing something than money. Of course I am happy to use both proprietary and libre software, but I believe Libre Source may be a positive thing in helping keep standards open and competition strong. MS Windows certainly improved dramatically back when MS realized GNU/Linux could become an actual competitor to them on the desktop. And I'm not sure OOXML would have been made an open standard without ODF and Open Office. Or perhaps it would but not till later.

      2. DropBear
        Facepalm

        Re: Pity they dropped Windows support

        They don't se code and software as a job or products, for them code *is* politics and a way to shape the world.

        Yup, because preventing other people to benefit from what you create unless they agree to hand over a chunk of cash they probably can ill-afford to spend is just sooooo much more noble than aspiring to let them enjoy it freely, with the hope of seeing that sort of thing happen more often in the future. Indeed. Terrorists, all of them...

    2. BinkyTheMagicPaperclip Silver badge

      Re: Pity they dropped Windows support

      Windows support is coming - they're currently working on it

  12. Stevie

    Bah!

    I cannot support this effort no matter how well-intentioned, as they needlessly broke the "libre name" paradigm established by the open office forkers a couple of years ago.

    It should have been SSL Libre of course.

  13. ofutur

    Someone has tried to use the library to compile a lot of commonly used apps and some of the results were shocking...

    "in getentropy_linux.c:

    extern int main(int, char *argv[]);

    #define HD(x) (SHA512_Update(&ctx, (char *)&(x), sizeof (x)))

    HD(main); /* an addr in program */

    the address of main() is used to gather entropy… very smart… NOT.

    most of the methods used in this file to gather entropy are very dubious.

    the crypto experts from OpenBSD should know better and just use /dev/urandom and/or getauxval(AT_RANDOM)

    instead of all these hacks."

    http://devsonacid.wordpress.com/2014/07/12/how-compatible-is-libressl/

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like