If they had contributed this patch when they first made it, then someone might have spotted the defect before the heart bleed bug was discovered...
Akamai has issued new SSL certificates to some of its customers after realising its customized OpenSSL was not immune to the Heartbleed bug as first thought. Some time ago, the web distribution giant modified the code to the open-source OpenSSL library and rolled the tweaked version out to just its servers: that adjustment …
Ah, but their special secret source made their product taste better. Or, .... not....
Contributed? Where to? OpenSSL does not take contribution outside of "their own", even critical bugfixes. And especially no security measures.
They clearly knew what the bug was, and had the options (in ascending order of complexity)
a) Disable the heartbeat service, which is basically never used anyway (until last week!)
b) Fix the bug
c) Do something to ensure that important things are always >64kB away from the memory space the server process might be using to respond to hearbeat requests
A config tweak vs a ~three-line fix vs writing a large chunk of control code to mess with memory allocation for certain functions. Which they then got wrong anyway.
Re: Er, why?
It's not obvious that they knew of this specific bug - developers were already concerned that OpenSSL's own "secret malloc sauce" was dangerous. Here's OpenSSH's Theo de Raadt gently remonstrating...
But yes, building OpenSSL with heartbeats disabled would have been good - unless of course they tried and found that this conflicted with some other config macro they needed, since most of the combinations weren't being built, let alone tested. Such a minority interest item shouldn't have been enabled by default anyway, especially in a security layer.
Re: Er, why?
"It's not obvious that they knew of this specific bug - developers were already concerned that OpenSSL's own "secret malloc sauce" was dangerous. Here's OpenSSH's Theo de Raadt gently remonstrating..."
How many times do I need to repeat myself?
This bug is a buffer over-read.
Nothing to do with malloc.
No malloc/calloc/jemalloc/magic-pixies-malloc would have helped.
Yeah, guard pages and canaries could help, but as it stands, so long as the memory being overflowed to still belongs to the process, there won't be any sigsev crash.
For a big outfit Akamai is doing better than feared
How often do we get to see a corp boast in public, get called out on it and then promptly eat that humble pie, discuss their mistake in reasonable technical detail, and set about digging its customers out of the mess? A couple of years back I'd have expected a promptly hurled spurious law suit (DCMA protection device violation, libel, damaging customer confidence, disclosure of trade secrets, freelance subversion...)
Maybe it's the influence of Pwn to Own (etc), or maybe they're both symptoms of an overall change, but it feels like our industry is growing up just a bit.
Re: For a big outfit Akamai is doing better than feared
Also, that is a very nice example of play between corporate and open source, working exactly as intended. Many eyes etc. ... benefiting almost everyone (NSA excluded).
The more I learn of OpenSSL the more I wonder about the the idea of
"Release a core set of functions, then role out more over time (or prioritize heavily used ones first)"
It seems if they had left the heartbeat message handling till later they might never have done it at all.
Prime example of a lousy open source user - makes improvements to critical widely used software - doesn't bother to feed back into the project. Worse considering how little support openssl seems to have been receiving despite how important it is. Oh well.
With freedom comes responsibility.
"Nothing against Akamai, but seriously: they held off replacing certs because they thought they were secure? Ugh," said Matthew Green
Umm.. a bit like everyone using OpenSSL then? If you believe you're secure, why would you replace certs that are not threatened?
Full marks to Akamai for acting responsibly when the flaws were pointed out.
Could be worse
Better than GIS software maker ESRI. They have issued a statement saying:
"ArcGIS Online – No customer action is required as mitigations have been applied to all service endpoints and certificates are being re-issued across the platform as a precautionary measure."
So they claim to have patched their servers but by 16/04 they still haven't replaced certificates and they are telling customers they don't need to do anything when a password change will be required. The mind boggles at the incompetence of such a large company with so many government and military customers.
What's wrong with this picture, and the general response to Heartbleed? Our servers are running software which may leak your private data. But *we will keep them running*.
In the contest between security and revenue, revenue wins.
Re: What's wrong with this picture,
The problem with taking ALL those servers offline to fix the issue is that the ensuing financial chaos would have made the recent banking/mortgage crisis look like a day in the park.
This is the fundamental problem in security: Does fixing the known security issue cause more damage than running with it? If so, how much more?
You need to mitigate, but the mitigation can't be worse than the disease. It's why although problematic, the secret notification people have a valid point about vulnerabilities. And if vendors are being responsible are the first route to take.