Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth? Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a bombshell Bloomberg report today. The story, which has been a year in the making and covers events it says …
None of the actors can be taken at face value, particularly on a story with obvious national security implications. That said, I have a bit more confidence in the statements of Apple and Bloomberg than the others (even if they do contradict each other).
And let's not forget that the US was caught engaging in this sort of thing with Cisco equipment being shipped to the middle east, so we also can't rule out that the devices were installed, but it was done by or on the behalf of the US government.
So, my take is simple -- we don't have enough information to make any kind of judgement about who did what, if anything, here.
#1. The Reg article missed a few things. For starters there were Bloomberg companion reports covering the thorny issue of outright-denial regarding the risk of potential Securities-Fraud:
https://www.bloomberg.com/view/articles/2018-10-04/computer-spies-hacked-reality
____
#2. Some of the chips were better hidden. Also there might have been more controls and tests on the critical chips, making it more risky to package them as a single integrated circuit, plus its more costly: "In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says."
____
#3. Who stands to gain from the publicity, who stands to get hurt? My own take... The US has done similar sneaky things so its almost certain that China has. The difference is, China has exceptional domestic leverage and could interfere at source, whereas the US doesn't have that level of control. But overall being a huge national security issue and potential shock to the US Cloud industry, its almost certain big tech would have been told to keep quiet and indemnified from any markets issues. But I also believe some elements in the US Govt wanted this to come out now that there's a Trade War... It might cause an Offshoring of manufacturing away from China, which could severely hurt.
Apple issued another denial that specifically said they aren't under any national security gag orders. I suppose if you were under a national security gag order capable of making you issue denials, it could make you issue a "we aren't under a gag order" denial.
Given the report that this was closely held within Apple, maybe when Apple was reached for comment and they internally contacted "people who would know about this" they simply didn't reach the people who did know. That is, Apple issued a denial because as far as they could tell, the denial was true. If a few engineers find something like this and report it to their manager, who says "let's take it to the FBI" and the FBI says "please don't talk about this with anyone else" it only gets as high as that low level manager.
IMHO it is quite plausible that when Apple spokespeople were contacted for comment, no matter how thorough they were in looking for any evidence that this story was true, they can't talk to everyone in the company so they might simply have not talked to the right people. The question is, if true, would those right people see the story in the press and that Apple has issued and decide to tell their higher-ups so Apple gets the story straight? Or would they keep their mouth shut, and figure correcting the record now will only make things worse?
"Apple issued another denial that specifically said they aren't under any national security gag orders. I suppose if you were under a national security gag order capable of making you issue denials, it could make you issue a "we aren't under a gag order" denial."
If Apple were issued a National Security Letter, under US Federal Law it would be a criminal act for them to admit it or to admit anything the lettered covered. So Apple claiming they aren't under any national security gag orders is meaningless. Such a denial is issued for the benefit of the dim witted.
"If Apple were issued a National Security Letter, under US Federal Law it would be a criminal act for them to admit it or to admit anything the lettered covered. So Apple claiming they aren't under any national security gag orders is meaningless. Such a denial is issued for the benefit of the dim witted."
But you cannot be ordered to lie, Apple would be well within their rights to either say that they are unable to comment or simply not put out a statement at all.
I suppose by drilling a somewhat larger via into one of the internal layers you should be able to hide a chip in the cavity it creates without any bulges whatsoever. Yes, it would be noticeable if you examine the board structure carefully (ie. knowing what you're looking for, by x-ray / transparency) but probably not on a cursory looking-at-the-external-layer-only check.
And I'm not sure such a method would give away anything extra - even if your chip is on the outside, mounted on the surface as any other chip, it would be clear from the PCB layout / trace / footprint modifications necessary to mount it that it was a job done at the factory, while such an extra component would be much easier to notice than an internal chip even on a less thorough check of the PCB.
The only way to plausibly deny the source of the modifications would be to bodge in the extra chip rework-style (possibly even requiring extra patch wiring) but that would stick out like a sore thumb at any kind of glance to anyone opening the case - and even that would merely shift the blame away from the factories themselves, still leaving China as the only logical suspect, unless someone tried blaming the NSA & co.
So if Apple are not under a national security gag order what they should do is issue a statement saying that they ARE under such an order.
They could only legally make such a statement if there was no gag order. Thus they would prove that they not in fact under such an order...
While you can't be ordered to lie, you can be ordered not to disclose information. This leaves you with the following options:
Apple: No comment.
El Reg Readers: So clearly it's happening.
Apple: Definitely not. We can categorically deny all of this, in any terms you like. Just read out sentences and we'll tell you that it didn't happen, to avoid any sense of our being disingenuous.
El Reg Readers: It's almost certainly not happening.
Judge, 2022: The government finds for the plaintiffs, owing to clear falsehoods released by the defendant in an attempt to protect them from adverse actions on their share price... [until you fall asleep]
Apple: We can tell you that we aren't under a gag order, and that we haven't found a security device embedded in supermicro servers we purchased between the dates of ... [and other overly specific terms]
El Reg Readers: They sound somewhat confident. Maybe we'll believe them, but we're not entirely sure.
Meanwhile, if there really is no chip and therefore no order, you have the following options:
Apple: No comment.
El Reg Readers: So clearly it's happening.
Apple: Definitely not. We can categorically deny all of this, in any terms you like. Just read out sentences and we'll tell you that it didn't happen, to avoid any sense of our being disingenuous.
Apple attorneys: Yes, this didn't happen, but if you are that specific, someone could find a loophole and get you to say something that we could get attacked for. We don't have the time to evaluate any specific statements, so we should just issue our own denial, as specific as you think it needs to be.
Apple: We can tell you that we aren't under a gag order, and that we haven't found a security device embedded in supermicro servers... [extra details to assure people watching that they're being honest and really trying to demonstrate that there is no cause for worry]
El Reg Readers: They sound somewhat confident. Maybe we'll believe them, but we're not entirely sure.
What effect? Both Amazon and Apple had their share price fall the past couple days, but it doesn't appear to be related to this article, since the NASDAQ as a whole fell more than Apple did and some stocks like Netflix fell more than twice as much. You could argue "the tech industry fell over worries these attacks might be widespread" but why would anyone have that worry about Netflix? Is someone going to care that China finds out what kind of movies they like?
I'd argue that the story actually makes Apple and Amazon come out looking really good. They detected the attacks quickly, when they were isolated rather than widespread throughout their infrastructure, and they acted immediately to get rid of the compromised hardware. How many other companies would have even figured this out? Think about how often you read about companies that have had hackers inside their systems for months if not years undetected - and it is FAR easier to find software nasties in your systems than a tiny component the size of a pinhead on your server boards. I mean, there's a whole selection of software designed for identifying and neutralizing malware, but you're on your own finding spy hardware.
You can't be court ordered to lie in the US as that would violate the 1st amendment.
I doubt they'd need to be ordered to lie. If the episode is real, all companies and governments concerned will want to hush it up. All the TLAs need to do is co-ordinate and participate in the denials and offer the companies concerned a guarantee that they won't be investigated or prosecuted in relation to any such denials.
"You can't be court ordered to lie in the US as that would violate the 1st amendment."
"You can't *LEGALLY* be court ordered to lie in the US as that would violate the 1st amendment."
TLAs have a certain attitude towards legal restrictions that get in the way of what they want to do, and often courts seem to indulge them.
That is not how it works with regard to the 1st Amendment. It protects speech that is for political advocacy from government interference. However, national security orders (FISA?) for businesses would not be in that category. (The other side of that coin is how Twitter, Facebook, etc can get away with suppressing Alex Jones "speech - they are private entities).
Embedding things in the PCB is really, really hard and the PCBs are made in a different facility to the assembly. Firstly it would be really hard to embed something in the substrate and require a completely different process at the facility it was made in. Then It would be totally random where those boards were inserted and they would possibly have to bypass the normal quality checks on arrival. They could only be targeted if multiple people along the supply chain were involved in the conspiracy and being coordinated.
The report states that there was " one version" with the PCB-layered chip, not all.
That would make one wonder about the expertise, resources, and authority required to vary the modifications in such different ways, and why those differing techniques would be chosen among.
Query regarding embedding chips in the motherboard substrate. Is this even a part of the normal manufacturing process? If it isn't, we can probably discount that part of the story as hyperbole.
That's roughly what I was thinking. Embedding a physical device, no matter how small or how smart, is such absolute proof of where the attack was carried out that it seems far too clumsy, and far too likely to be found out.
Still, as others have pointed out, none of the actors in the story necessarily inspire one with confidence of truth, while all have something to gain from being manipulative.
But overall, this sounds more like the current equivalent of a Red under the Bed hysteria which seems so boringly cyclic in some parts of the world, the Chinese being the bogeyman-de-jour. Where's Arthur Miller when he's needed?
With silicon thinning (already used for HBM stacks, for example), you could easily stick the silicon into the motherboard substrate, between standard layers. The bulge would be imperceptible, and the thin silicon might not register for x-rays or other hardware scanning solutions.
I presume this chip is installed onto a serial data link to the flash memory, and on power on it intercepts the serial bitstream from the flash, and adds enough to install its payload.
The hardware security solution to this is on-board flash and memory on the server management processor, preferably on the same die, made with security hardening techniques.
"The hardware security solution to this is on-board flash and memory on the server management processor, preferably on the same die, made with security hardening techniques."
Thus forcing them to replace that chip, find a way to bypass it, or to corrupt it in some way.
Interfering with the manufacturing process, with sufficient technical skill, seems to be almost unstoppable. Any 'solution' can be obviated, bypassed, or removed.
"Query regarding embedding chips in the motherboard substrate. Is this even a part of the normal manufacturing process?"
This is not a common thing to do, as it increases manufacturing costs, but it certainly has been done (for legitimate, not sneaky, reasons) before, so it's something that is a manufacturing option.
It's not part of the normal manufacturing system for conventional boards, though there are high-density stack-chip manufacturing methods with similarities. I think it could be done, at considerable cost and inconvenience. So unlikely in high volume, but possible for 'specials'.
The embeding of active/passive components can be traced to the 1980's... and IBM published a paper
( https://www.jstage.jst.go.jp/article/jiepeng/2/1/2_1_134/_pdf/-char/en )
<QUOTE>
Embedding components inside a PCB motherboard or a substrate provides literally a new dimension to achieve the needs of today’s high end electronics manufacturing. Component embedding inside a substrate is not a completely new idea, and several technology approaches have been in development over the years – the first real attempt to commercialize an embedding technology was done by GE in mid 80’s.[1] But only now has the market evolved to accept component-embedding solutions and at the same time the infrastructure has matured to a level where component embedding becomes a commercially viable solution.
</QUOTE>
so there are no technical obstacles...
Yes, it can be done. Inserting resistors and diodes is common; inserting chips and capacitors I've only personally seen once. But then I worked in the software side of a defense company, so I had little exposure to this stuff. Search on the website of a company like Curtis Wright and if you can find a photograph of a processor board without metalwork, then you are likely to only see large BGA's and a few interface chips (the devices that need the cooling) and an otherwise bare PCB surface (to allow a ground plane on each side of the PCB stack to cut down on EMC.)
Your real question should be can it be done cheaply enough, with the follow up of what is the budget for this information?
Your personal take of a false flag by the US is laughable and not even worthy of a B movie (direct to hulu) or something like that.
Look, Trump's beef w China and tariffs are more than just trade. He wants to apply pressure on NK.
At the same time... you don't even think about China's activities surrounding their man made island which they now have put military forces on and are claiming ownership over some oil fields that are supposed to be owned by Viet Nam (IIRC).[Note: I could be wrong about the other countries involved... going from memory]
I'd say that Bloomberg's reporting seems to be accurate consider ancillary factors going on.
Do you really bork a bunch of hardware over a firmware upgrade that has malware associated with it? Or do you just upgrade to a fixed release? Or go back to a prior release?
There's more, and what's interesting is which motherboards... blades.
But overall being a huge national security issue and potential shock to the US Cloud industry, its almost certain big tech would have been told to keep quiet and indemnified from any markets issues.
And a fairly large segment of civilian US Government agencies and projects operate on AWS platforms. Such as the compromised more than once NFIP, aka NVITS NFIP Virtual Information Technology System (NVITS) (NFIP being National Flood Insurance Program). I could mention more, but at a severe cost, due to an NDA. As NFIP is going to close down, due to a lack of congressional funding, that NDA is entirely moot, as is flood insurance in the US.
There seems to be a general rule of thumb that when US intelligence departments leak alarming stories via compliant press contacts, it's usually the case that the US is already doing this themselves and are sweating buckets over the thought that someone else might be doing it as well. We saw exactly this in the run-up to the Stuxnet reveal, and we saw exactly this in the backdoors being installed in Cisco networking equipment.
I remember the same sort of vague but alarming stories claiming that foreign powers were infiltrating SCADA systems and could use that do destroy utility equipment. They even built a lab type setup of a diesel generator with attacked SCADA system and demonstrated it. Meanwhile the utility industry scratched their heads in puzzlement, because despite the alarm and panic in government, industry couldn't pry any actual details out of them so they could take preventive action and nobody was seeing it in the wild. And then the Stuxnet story came out and we found out the panic was about how the US (with the assistance of Israel) had infiltrated the SCADA systems controlling Iranian enrichment equipment and was using it to conduct sabotage and the US were afraid they would be hacked back.
To go back to the mysterious motherboard chips, if this was real, I would expect someone to present actual hacked hardware along with demonstrations of what it did. After all, if the story were real then it's not like Chinese wouldn't already know everything about it, so what's the point of hiding it?
And Amazon's and especially Apple's denials are pretty strong. If they were obfuscating the issue, then they would just release their usual vague waffle.
I suspect this story is complete bullshit. The use of a security company in Ontario Canada is also very interesting. At this very moment the US is putting lots of pressure on Canada to try to get them to ban Huawei equipment from important Canadian networks. It would not be surprising if this whole story were to be an exercise intended to pressure allies into stepping into line behind the US in freezing Chinese tech companies out of western markets in favour of equipment that has the backdoors of "friendly" countries in it.
The CIA Headquarters are in Langley, VA, which is outside McLean. MITRE is in McLean, though, as are many other organizations that one might expect to be providing support to the government. It is far more plausible that a meeting with a number of industry executives would be held off-campus than at an agency HQ, even if the host of the meeting was an agency (a meeting at a think-tank could be about anything, so the security issues are easier to handle).
My biggest problem with Kieran's article, is that it definitely needs the Touch of an Editor, because there are three sections with Purpose Creep.
"Cleverly" written, attributable to "condensing", but ultimately Really Bad Language.
Given the report, there will be an army of BOFH's ripping out MB's and minutely inspecting them, so we'll know soon enough. But K. is guilty of "tendentious reporting" , to use the polite European term.
Sounds more like you just disagree with Kieren's points. We spent pretty much all day putting it together while everyone was either rushing in to cut'n'paste Bloomberg or Apple/Amazon/SM's denials.
Instead, here's a technical breakdown of the matter at hand, looking it from both sides.
If you don't like it, Zzzzzdnet's that way -------------->
C.
Given the report, there will be an army of BOFH's ripping out MB's and minutely inspecting them
As one of those BOFH's, i'm going to comment that there is no earthly point doing that for >99.9% of BOFH's. To make it worthwhile, you'd need:-
1) The original plans sent to the fab.
2) the ability to check the motherboard for objects that shouldn't be there that are on the nanometer scale.
In addition, after you've found something that you think might not be there then you'd need:-
3) the ability to figure out what the hell things are down to a scale of ~50nm. Xray scanners are not particularly common, and most of those aren't going to resolve down to the level where you can recognise components inside a chip, let alone allow you to identify them and spot things that have been added to the original design.
4) the ability to pull the embedded code off microchips to figure out what they are doing is as per the design.
Yeah, um. Next to nobody has #1, and I suspect the number of teams with the ability to pull off 2 is very, very limited. 3 & 4, um. I'm thinking "count them on your fingers".
"the ability to check the motherboard for objects that shouldn't be there that are on the nanometer scale."
So, um, you reckon they did all that firmware hijacking via a single flip-flop...? Because double or single digit nanometer scale is what individual features of a single transistor are at, not any fucking chip of any fucking level of complexity.
So, um, you reckon they did all that firmware hijacking via a single flip-flop...? Because double or single digit nanometer scale is what individual features of a single transistor are at, not any fucking chip of any fucking level of complexity.
If we are being paranoid enough to check this stuff, wouldn't we be paranoid enough to check that what is in the chips matches what we expect to be in chips?
If we are being paranoid enough to check this stuff, wouldn't we be paranoid enough to check that what is in the chips matches what we expect to be in chips?
In some industries, we do that all the time because of component counterfeiting (a major problem for kit that needs to be available for 20 or more years),
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
