back to article Debian on track to prove binaries' origins

Debian is on its way to becoming what could be the first operating system to prove the origin of its binaries, technologist Micah Lee says. The feat will allow anyone to independently confirm that Debian binaries were built from a reported source package. So far a project team devoted to confirming the reproducibility of …

  1. Electron Shepherd
    Boffin

    You also need to be able to trust the compiler...

    ... and the compiler of the compiler...

    Historically, back doors have often lurked in systems longer than anyone expected or planned, and a few have become widely known. Ken Thompson's 1983 Turing Award lecture to the ACM admitted the existence of a back door in early Unix versions that may have qualified as the most fiendishly clever security hack of all time. In this scheme, the C compiler contained code that would recognize when the login command was being recompiled and insert some code recognizing a password chosen by Thompson, giving him entry to the system whether or not an account had been created for him.

    Normally such a back door could be removed by removing it from the source code for the compiler and recompiling the compiler. But to recompile the compiler, you have to use the compiler — so Thompson also arranged that the compiler would recognize when it was compiling a version of itself, and insert into the recompiled compiler the code to insert into the recompiled login the code to allow Thompson entry — and, of course, the code to recognize itself and do the whole thing again the next time around! And having done this once, he was then able to recompile the compiler from the original sources; the hack perpetuated itself invisibly, leaving the back door in place and active but with no trace in the sources.

    Taken from http://catb.org/esr/jargon/html/B/back-door.html

    1. Message From A Self-Destructing Turnip

      Re: You also need to be able to trust the compiler...

      Stop it! My brain hurts now.....hmmm I wonder if anybody has managed to back door the abacus.

    2. JimmyPage Silver badge
      Boffin

      Fuck that - what about the silicon ?

      You could code your first assembler (F1 ?) in raw machine code. But it would still be compromised if the actual architecture of the CPU had been nobbled.

      And with daily revelations about what the NSA/GCHQ have been up to, it's not impossible (and the likeliness is certainly non-zero) that CPU instruction sets have been hacked. After all, has anyone checked what 27 NOPs in a row *really* do in the latest Intel offering ?

      I have to smile when people make a big fuss about open source, while ignoring the chipsets you're running that source on.

      1. Electron Shepherd

        Re: what about the silicon ?

        Agreed. It seems unlikely that given what else they've been up to, the spooks haven't attacked the low level stuff. A modern motherboard's BIOS is upgradeable, and the BIOS has the capability to patch the CPU's microcode. Put those two together and you have an attack that will patch the CPU itself, and will survive a hard disk reformat. It's more likely that the attack would just be to patch the boot loader so that it patched the OS loader before the operating system starts, but in principle, modifying how the CPU itself behaves is perfectly possible.

        1. RobHib
          Thumb Up

          Re: what about the silicon ?

          "Agreed. It seems unlikely that given what else they've been up to, the spooks haven't attacked the low level stuff.

          I've no idea whether or not spooks have attacked the low-level stuff but I wouldn't be a bit surprised. Another obvious and almost guaranteed method of attack on low-level hardware and firmware will come from industrial espionage targeted by commercial operators (industrial spying). Then there are governments with plans for secret and or mandatory backdoors into smartphones. However, for the protagonists to effectively implement or render any of these schemes into silicon they would need to be well resourced and well funded.

          This isn't entirely idle speculation either. No specifics, but the matter of whether certain important security hardware had been 'gotten at' and had 'suspect chips' came across my desk about 15 years ago. In IT terms, that's eons ago. If the issue arose with me that long ago then you might like to speculate what's transpired since. I'll leave you to guess. It's ludicrous to think that back then that we alone were the only ones who were considering such matters—of course not. Furthermore, even back then such ideas were far from new (had they been I'd like to think I'd have patented the concept). ;-) (BTW, a while back, I was laughed at in these posts for having the hide and temerity to even suggest such things were possible with silicon. Oh dear.)

          At the time, we were concerned about the possibility of specially designed Trojan components, ICs, ASICs etc., with 'modified' silicon being incorporated into systems and equipment—devices that had normal part numbers together with the usual physical characteristics etc., ones that worked in the usual standard/specified way so as to fool investigators but upon receipt of secret commands would also perform 'extra' undocumented (hidden) tasks.

          Your post about Ken Thompson's 'doctored' compilers together with that of JimmyPage about Silicon, immediately reminded me of issues we'd actively investigated years ago, specifically backdoor techniques can also be incorporated in or patched into silicon compilers—consider it, its ramifications are just humungous. At the time the matter arose, our principal concern wasn't the exact specifics of how Si or other components were modified or could be altered, rather it was how to go about detecting 'dodgy/suss' ICs, and other components—or even complete subsystems or modules that may be fully encapsulated but suspect (does one test or break open a black box?). And, I can assure you, it ain't easy!

          Since then I've examined the matter at some length, Si compiler hacks are feasible and doable. That said; don't immediately jump to paranoiac conclusions that it's happening everywhere. Essentially, it'd be a large operation that'd need resources of such a magnitude that only governments and large corporations could command (it'd also need complicity between government and foundry).

          Nowadays, users have little or no control over BIOS/flash ROM as they once* did, instead HW and SW manufacturers have combined to essentially lock users out of that part of their PC, they've now little more than token access. Similarly, there is considerable potential for 'compromised' chips, ASICs and other 'modifiable' hardware to compromise security completely. PCs are not alone; many types of tech gear continue to be highly vulnerable security risks as they too have BIOSes, and as technology becomes more dense, the problem is only going to get worse. Instead of manufacturers closing loopholes, they keep them open or even widen them for their own convenience. BIOS hacks and patches (such as the patched ROM hack that you're likely to have on your laptop this very moment), are easily installed by both manufacturers and hackers alike but the reverse is seldom true for users.

          * It's not an exaggeration, I used to compile the assembler BIOS source in my Godbout CompuPro. It gave me incredible control over the system, it's control I do not have in the PC ;

          So who's the true beneficiary now that your mobo's BIOS is easily hacked and controlled? Right, it's not the user (as fewer and fewer tweaks are now being made available in the BIOS UI of many modern machines—fewer tweaks mean fewer help-desk calls, but it also means many machines are running sub-optimally.) Not only is control being continually whittled away bit-by-bit from users in the BIOS but it's also happening in Windows. With every new release, users have increasing difficulty in accessing the 'low-level' areas within in Windows (it literally takes me weeks to tailor Window the way I want it. It sucks, and it ought not to be necessary.)

          This begs the question about who's ultimately responsible for security breaches when manufacturers deliberately remove control from users. For example, once mobos had 'hard' DIL switches that couldn't be tweaked by Windows or remote hackers, now there's no physical DIL switches at all, as MS and others forced their removal only to have them replaced with 'soft' switches—switches Windows can now actually command. (And, in many instances, that's worrying). Moreover, why do hard disks no longer have a mechanical write-protect switch as they once had?

          The ever-increasing 'soft' control over vast amounts of our new (and older) technology is becoming a serious problem. For example: once a sluice gate on a storage dam had to be opened manually, now, more often than not, it's opened by software from some remote terminal, and (stupidly) there's no human to hand if something goes seriously wrong. Matters are then made even worse when this 'improved' now-more-fragile system, becomes the subject of threats, hacking etc. We've now instances where critical infrastructure has becomes more vulnerable simply because of the actions of idiots who want to remote control a previously intrinsically reliable system that's worked for decades—solely on the grounds that it can be done (the excuse being that it's cheaper)!

          What happens next would put a circus to shame, the government and its band of woolly thinkers have to tighten security laws around critical infrastructure because they've weakened it—and in the end it'll cost a damn side more to fix, than if they had initially left it alone. In the world of engineering in which I grew up, we would deem this madness. More accurately, we're now living in a world where the digital addiction meme is more contagious than the influenza virus.

          It doesn't require much stretch of the imagination to figure where the next step leads: mass government surveillance has come about, not because it makes sense but because it can be done easily—put it together with the fact that people have fallen in love with the technology and the woolly thinking becomes even woollier, as they're no longer thinking rationally (the masses falling in love with their tools is, historically, a unique phenomenon—in the past tools meant work).

          Like the Emperor's new clothes, almost everyone's now caught up in a wave of general madness, either because they want to see the 'clothes' or they don't want to admit that they can't for fear of embarrassment. Correct, this is crowdsourcing gone crazy! Plato covered this in the Republic millennia ago—when you're sick, go to best advice available: a doctor; only a bloody fool would prefer to take the alternative option, that of asking the LCD-opinion of a crowd.

          Ipso facto, we're now more vulnerable to government snooping, violations of privacy by manufacturers, infiltration by serious hackers and of being exposed to data and ID theft, now it's clear why. Moreover, there seems little progress through the impasse: technology is progressing leaps and bounds, yet seemingly simple problems such as using it are fraught with difficulties. Deep-down, as years of experience attest, Microsoft et al truly do not want the great unwashed in full control, we're simply not trusted—and because they didn't listen to us, that's exactly why we've ultimately ended up with abominations like Window 8.

          All this ought to be of considerable concern to users, but I see little in the Big Picture Department except inaction. (Somehow, there's still a widespread belief that Microsoft etc. knows best. It's utter bullshit of course.)

          Hope I'm wrong but I reckon the security issues will likely soon reach the stage where the security at the bottom (physical) layers of the OSI model will be sufficiently compromised that business communications over the internet will degenerate into a hit-and-miss affair—or shroud that be 'mess'. Embedded and other systems, who've had it good until now will also be in for a similar shock. Pessimistic perhaps, but I don't think so. After all, if it were possible to equate dissimilar standards, what other industry could be so degenerate as to have the equivalent standards to that of IT security? Security so poor that the NSA, GCHQ, DSD etc. look into our private lives with such ease that it's as if they're looking through transparent glass; then, almost every day, El Reg has spectacular stories such as the heist of millions of credit cards; and then there's the almighty Microsoft patch saga: year after year, decade after decade, MS continues to issue thousands of patches for its mouldy, over-bloated, holier-than-Swiss-cheese operating systems, and yet it still has the audacity and unmitigated hide to pontificate that its OSes are state of the art.

          It's hard to deny the IT Industry has sleazebag ethics that'd put used car salesmen to shame.

    3. Lee D Silver badge

      Re: You also need to be able to trust the compiler...

      And as the same guy - and others - have since described, diverse-double-compilations solves this by detecting any difference produced by the compiler itself.

      https://www.schneier.com/blog/archives/2006/01/countering_trus.html

      You just get to a point where you have a known-good hash for a compiler that compiles consistently every time, compliant to the word of its source code, detecting if you're even using such a "malicious" compiler.

      From that point on, you just make sure you only use that compiler for the rest of the checks.

  2. A Known Coward

    What a complete joke

    Debian packagers have a terrible reputation among software developers for introducing their own changes to applications, often breaking them in the process. In some cases introducing spectacular security holes (OpenSSL). These packagers can make changes at will to the actual code you'll use despite never having to prove their programming abilities to anyone, bypassing the stringent hiring processes, code review and testing of software projects.

    So this work 'proves' that the source they built with was the source code they include in their source packages, but it doesn't prove the code in their own source packages matches up with the original source code released by the application developers. How many people are going to diff the packaged code against the original code to be sure that no adverse, or malicious, changes have been introduced?

    I'd be far more impressed if they were introducing something which proved that their packages were built from completely untampered code and not their own dodgy reference copy.

    1. localzuk Silver badge

      Re: What a complete joke

      If you really want to check things like that, you have to check it yourself. The old adage "you can lead a horse to water, but you can't force it to drink" comes to mind.

    2. Daggerchild Silver badge

      Re: What a complete joke

      That sword cuts both ways. Yup, they klutzed an initialiser in openssl. They ain't doing that again anytime soon. But they also patch things that haven't been patched upstream yet and otherwise fix stupidities. In fact, on balance, they're firmly in the positive on making smart changes by my reckoning.

      In fact, I dismantle the debian source diff and browse their patches before compiling any stuff on any platform. It's a good source of vitamins and iron.

      Also "bypassing the stringent hiring processes, code review and testing of software projects" : sorry, experience demands that I snigger here :)

      If it worries you that much, volunteer your own time to debian and become your own package maintainer. Danger: May involve work and responsibility.

      1. A Known Coward

        Re: What a complete joke

        Sorry, I'm too busy being one of those devs whose code they massacre to do their job for them as well.

        Becoming a packager involves responsibility? Now you're the one making me laugh, all those users who run into bugs with packages come straight to us, we're the ones who have waste time triaging their bug reports and send them back to the packagers who just aren't interested. When users encounter bugs in an application it's the original application developers who get the bad reputation even though those bugs were introduced at the packaging stage.

        It's reached the point where we now refuse support for Debian, and may soon refuse them permission to use our trademarks (Just as Mozilla did for Firefox).

        1. kwhitefoot
          Flame

          Re: What a complete joke

          > we now refuse support for Debian, and may soon refuse them permission to use our trademarks

          We who? Who's trademarks? If things are as bad as you say it seems a little churlish to withhold that information.

    3. SImon Hobson Bronze badge

      Re: What a complete joke

      > So this work 'proves' that the source they built with was the source code they include in their source packages

      Yes - that's it's sole function.

      > but it doesn't prove the code in their own source packages matches up with the original source code released by the application developers

      No, because you can do that yourself - just by diffing the sources.

      The whole point is that you can inspect the source yourself, and if you are bothered, you can compare the upstream and Debian versions. That's relatively easy as the sources are there - there's nothing you can't inspect.

      What you haven't been able to show, until now, is that the source Debian tell you a package is built from, and which you may have compared to the upstream source, is what was actually used to build the binary.

    4. tom dial Silver badge

      Re: What a complete joke

      Is there a Gnu/Linux binary distribution that is functional and does not modify any developer's application code?

      I leave it to a Debian developer to describe the entry path, but it certainly is not true that there is no monitoring of their actions or evaluation of their competence.

      1. Anonymous Coward
        Anonymous Coward

        Re: What a complete joke

        Well, that poster (A Known Coward) has got a point.

        > Is there a Gnu/Linux binary distribution that is functional and does not modify any developer's application code?

        I don't know. Maybe Slackware? In any event, the difference between Debian and other major distros is in what they change and how.

        Other major distros, generally speaking, consist of "community" FOSS and their own applications (not always FOSS, e.g., RHEL, SEL). For the "community" part, as part of the build process, often patches will be applied that either deal with incidental things like default location of configuration files or installation paths, or apply fixes to known bugs, with the relevant patches being usually shared with the upstream devs, who eventually incorporate them onto their own releases (although sometimes this may take a while, depending on that project's development workflow, policies, etc.).

        The difference with Debian is that in some cases the latter's packagers will just patch away stuff just because they don't like it. Case in point: Bash's /dev/tcp and /dev/udp port redirections¹. You may argue, as the Debian packagers have done, that that is a misfeature and should not be there or a different mechanism should be provided--that's fair enough², but if the upstream dev disagrees with you, I would expect you just live with it (remember, you are just the packager not the project leader) or if it really annoys you so much, you spawn your own fork: at least that way your users will be aware that there are changes. That is exactly what they were forced to do with Mozilla's browser and email client. I am no big fan of the Mozilla Foundation, but in retrospect they did do the right thing there.

        ¹ This is exactly what made me migrate away from Debian years ago, after I spent significant time and effort trying to troubleshoot a series of scripts that all of a sudden were not working. It had never occurred to me that some sorry fucker would take away functionality just because he didn't like it.

        ² Although the same argument could be applied to the various shell built-ins that replace actual system commands, such as "time".

      2. jake Silver badge

        @ tom dial (was: Re: What a complete joke)

        "Is there a Gnu/Linux binary distribution that is functional and does not modify any developer's application code?"

        Slackware is closer to that ideal than any other Linux distribution.

        Be careful with Slack, though. If you use it, you might actually learn something.

    5. Terry Cloth
      Linux

      Re: What a complete joke

      @ A Known Coward

      So this work 'proves' that the source they built with was the source code they include in their source packages, but it doesn't prove the code in their own source packages matches up with the original source code released by the application developers.
      Well, actually, it does. Back in the early days they patched the source to their specs, and distributed the patched source. Somebody noticed the very point you bring up, and now Debian source packages include
      • precisely the upstream source (in files named <package_upstream-version>.orig.tar.gz,
      • Debian's patches (named <package_upstream_version-revision>.diff.gz), and
      • other control information needed to generate the binary.
      So yes, you can prove their build is based on the upstream source. And their changes are available right there for your review.

  3. Andrew Commons

    I don't think this is what it is all about

    "The feat will allow anyone to independently confirm that Debian binaries were built from a reported source package."

    Back in about 1983 I managed a small group tasked, amongst other things, with distributing software to hospitals. Part of the QA process I introduced involved taking a release of the source code and a copy of the software used for acceptance testing. The next step was to rebuild the software from the supplied source and compare the results with the acceptance test versions. This was VAX/VMS and the undocumented CHECKSUM command was used for this comparison (CHECKSUM/IMAGE for the curious, it took out timestamps). If the two did not match it was all sent back to the developers with a 'please explain'.

    Roll forward 10 years after a period of developing security software for VAX/VMS (see NIST SP 800-6) and I found myself doing something similar but more focused on determining whether the software had been built from code in the source code repository, effectively whether it was built from the approved package. This required a more detailed analysis of the information stored in the executable images and has some similarities to the initiative reported here.

    So, I've done some of this stuff and have some idea of where it can come apart.

    (1) Change your tool chain (even different version of the same tool chain) or architecture and it will all probably break. The Alpha VMS compiler/linker tools did not dump the same sort of information into the binaries and it largely invalidated the VAX/VMS tools I had developed.

    (2) Different build options, even when using the same tool chain and target architecture, will result in functionally equivalent but very different binaries.

    (3) Anything you find embedded in an image can obviously be manipulated so even if the same architecture and tool chain and options are used the contextual information must not be trusted without additional verification.

    For all of the above I think this is not the 'feat' you are looking for.

    Given some binaries and a source code package proving the link between the source code and the binaries is non trivial (again I have some form here having been tasked with reverse engineering the source code library that represented the executables currently in production...).

    What they are testing is the statement "It should be possible to reproduce, byte for byte, every build of every package in Debian.", this is from the WIKI reference a couple of clicks on from the referenced "report note".

    This has very little to do with the claim in this piece. If you have the same source code, and the same tool chain executed with the same options you should be able to prove that you have achieved the same results.

    Nothing like the headline statement.

    1. Bronek Kozicki

      Re: I don't think this is what it is all about

      My thinking exactly. If you need to have exactly reproducible setup, build from sources yourself (in case of Debian, IIUC that involves dpkg-dev ) and maintain your own repository, otherwise such guarantees are rather pointless

    2. Anonymous Coward
      Anonymous Coward

      Re: I don't think this is what it is all about

      "If you have the same source code, and the same tool chain executed with the same options you should be able to prove that you have achieved the same results."

      Should, but don't, because compilers, linkers, pre- and post-processors like to stuff timestamps, directory names, user names, and randomness (for instance, if a multi-core build finishes compiling files in a different order, or if a compressor or archiver is passed a file list in a different order), all over the packages they're creating, which will differ between machines, times, and runs.

      The whole point of the project, is to come up with a toolchain and automated build processes that don't do this, so that someone can download the source tree, build it, and end up with a binary that is identical to everyone elses.

  4. Primus Secundus Tertius

    Trusted code

    I have sometimes wondered whether one of the shadowy governemnt agencies maintains code that it can trust. And, indeed, as Jimmy Page notes, whether they have chips they can trust.

    In Britain, possibly not, because it all costs. In the USA, perhaps they do.

    1. Anonymous Coward
      Anonymous Coward

      Re: Trusted code

      > And, indeed, as Jimmy Page notes, whether they have chips they can trust.

      Certainly not in my local, I wouldn't trust anything coming out of that kitchen.

  5. Anonymous Coward
    Anonymous Coward

    mm?

    Isn't the main reason backdoors and issues get into the FOSS executable that people aren't interested in looking at the boring everyday bits of software (openssl, bash, etc) and instead are busy looking at the latest bleeding edge toy or designing yet another windows manager for linux?

    1. Anonymous Coward
      Anonymous Coward

      Re: mm?

      That's life, my friend. I don't think anyone here has mentioned OpenBDS (except for deriding their efforts) because people don't seem to care much about security, especially these days.

  6. channel extended
    Linux

    Open Source Code

    There is one distro that always compiles from source code.

    GENTOO!! Stage one tarball if you've got the nerve.

    Face it, if it's that important you do the compiling yourself. Yes it takes more time for an install, but you get what you compile.

    1. jake Silver badge

      @ channel extended (was:Open Source Code)

      Out of curiosity, where did you get your compiler binary?

      http://cm.bell-labs.com/who/ken/trust.html

      1. channel extended

        Re: @ channel extended (was:Open Source Code)

        I wrote, in 1988 approx., my own c compiler. Yes YACC in z80 asm. Then when I learned of the Gentoo distro I rewrote for x86 and tried to compile. I will say there is a learning curve and probably would not do something like that today (no time). I learned to write 6809,z80,and 6502 asm when the x86 was being born.

        My opinion - Intel and IBM screwed us all with the 8080, Motorola had a much better, technically, chip with the 68000.

        1. jake Silver badge

          Re: @ channel extended (was:Open Source Code)

          You didn't answer my question, Channel Extended. Where, exactly, in your case, compiling Gentoo, did you get your compiler executable? To say nothing of the linker and assembler.

          Kids these days refuse to see the woods for trees ...

  7. Anonymous Coward
    Coat

    Tip of the iceberg

    Even the C guys will admit it compiles too damn slow; Go doesn't. GCC has always been slow, but autoconf/autocrap is far worse. Linux+package build times could be maybe 10-100 times faster just by switching to cmake.

    However, that won't change the fact that Linux/Unix toolchains/libs/environments are cumbersome and bloated. It'll never be as frictionless as say JS, it'll never be ported to a better language (not even a modestly-improved C), and it'll never be guaranteeably secure. It's the best thing going now, but it's a dead end, and as a sysadmin I'm afraid it's all going to unravel in the next few years (months?) in the face of increasingly sophisticated attacks.

    I would put my money - and time - on new OSes, languages, and hardware, all designed for simplicity and maintainability and uniformity. And that's for a secure communication platform only, not fun stuff - you can't have it all.

    1. jake Silver badge

      @ tnovelli (was:Re: Tip of the iceberg)

      "Even the C guys will admit it compiles too damn slow"

      HELL no! I'd rather be thorough than fast ... ask my wife ;-)

      During the meanwhile, a dead-end? I'll bet you used bubble-gum that your telephone, your television, your car, and any other non-Redmond-based product you use that requires an OS is *nix-based.

      "I would put my money - and time - on new OSes, languages, and hardware, all designed for simplicity and maintainability and uniformity."

      OK. Enjoy. (hint: today's hardware/OS/language combinations are not exactly what I would call "simplistic", most are not easily maintainable, and almost all are proprietary, and nowhere close to uniform.)

      "And that's for a secure communication platform only, not fun stuff - you can't have it all."

      My very secure firewall runs BSD. Just like Cupertino's not-very secure commercial iThingies.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like