back to article Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?

Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a bombshell Bloomberg report today. The story, which has been a year in the making and covers events it says …

  1. JohnFen

    My take?

    None of the actors can be taken at face value, particularly on a story with obvious national security implications. That said, I have a bit more confidence in the statements of Apple and Bloomberg than the others (even if they do contradict each other).

    And let's not forget that the US was caught engaging in this sort of thing with Cisco equipment being shipped to the middle east, so we also can't rule out that the devices were installed, but it was done by or on the behalf of the US government.

    So, my take is simple -- we don't have enough information to make any kind of judgement about who did what, if anything, here.

    1. Anonymous Coward
      Anonymous Coward

      'None of the actors can be taken at face value, particularly with national security implications.'

      #1. The Reg article missed a few things. For starters there were Bloomberg companion reports covering the thorny issue of outright-denial regarding the risk of potential Securities-Fraud:

      https://www.bloomberg.com/view/articles/2018-10-04/computer-spies-hacked-reality

      ____

      #2. Some of the chips were better hidden. Also there might have been more controls and tests on the critical chips, making it more risky to package them as a single integrated circuit, plus its more costly: "In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says."

      ____

      #3. Who stands to gain from the publicity, who stands to get hurt? My own take... The US has done similar sneaky things so its almost certain that China has. The difference is, China has exceptional domestic leverage and could interfere at source, whereas the US doesn't have that level of control. But overall being a huge national security issue and potential shock to the US Cloud industry, its almost certain big tech would have been told to keep quiet and indemnified from any markets issues. But I also believe some elements in the US Govt wanted this to come out now that there's a Trade War... It might cause an Offshoring of manufacturing away from China, which could severely hurt.

      1. diodesign (Written by Reg staff) Silver badge

        Re: Anonymous coward

        "The Reg article missed a few things"

        Actually, we discussed the fact that the companies may have been ordered to lie.

        "Some of the chips were better hidden"

        We mention that, too.

        C.

        1. Anonymous Coward
          Anonymous Coward

          One thing that apparently happened after this story was posted

          Apple issued another denial that specifically said they aren't under any national security gag orders. I suppose if you were under a national security gag order capable of making you issue denials, it could make you issue a "we aren't under a gag order" denial.

          Given the report that this was closely held within Apple, maybe when Apple was reached for comment and they internally contacted "people who would know about this" they simply didn't reach the people who did know. That is, Apple issued a denial because as far as they could tell, the denial was true. If a few engineers find something like this and report it to their manager, who says "let's take it to the FBI" and the FBI says "please don't talk about this with anyone else" it only gets as high as that low level manager.

          IMHO it is quite plausible that when Apple spokespeople were contacted for comment, no matter how thorough they were in looking for any evidence that this story was true, they can't talk to everyone in the company so they might simply have not talked to the right people. The question is, if true, would those right people see the story in the press and that Apple has issued and decide to tell their higher-ups so Apple gets the story straight? Or would they keep their mouth shut, and figure correcting the record now will only make things worse?

          1. The Man Who Fell To Earth Silver badge
            Black Helicopters

            Re: One thing that apparently happened after this story was posted

            "Apple issued another denial that specifically said they aren't under any national security gag orders. I suppose if you were under a national security gag order capable of making you issue denials, it could make you issue a "we aren't under a gag order" denial."

            If Apple were issued a National Security Letter, under US Federal Law it would be a criminal act for them to admit it or to admit anything the lettered covered. So Apple claiming they aren't under any national security gag orders is meaningless. Such a denial is issued for the benefit of the dim witted.

            1. Spazturtle Silver badge

              Re: One thing that apparently happened after this story was posted

              "If Apple were issued a National Security Letter, under US Federal Law it would be a criminal act for them to admit it or to admit anything the lettered covered. So Apple claiming they aren't under any national security gag orders is meaningless. Such a denial is issued for the benefit of the dim witted."

              But you cannot be ordered to lie, Apple would be well within their rights to either say that they are unable to comment or simply not put out a statement at all.

            2. DropBear

              Re: One thing that apparently happened after this story was posted

              I suppose by drilling a somewhat larger via into one of the internal layers you should be able to hide a chip in the cavity it creates without any bulges whatsoever. Yes, it would be noticeable if you examine the board structure carefully (ie. knowing what you're looking for, by x-ray / transparency) but probably not on a cursory looking-at-the-external-layer-only check.

              And I'm not sure such a method would give away anything extra - even if your chip is on the outside, mounted on the surface as any other chip, it would be clear from the PCB layout / trace / footprint modifications necessary to mount it that it was a job done at the factory, while such an extra component would be much easier to notice than an internal chip even on a less thorough check of the PCB.

              The only way to plausibly deny the source of the modifications would be to bodge in the extra chip rework-style (possibly even requiring extra patch wiring) but that would stick out like a sore thumb at any kind of glance to anyone opening the case - and even that would merely shift the blame away from the factories themselves, still leaving China as the only logical suspect, unless someone tried blaming the NSA & co.

            3. Anonymous Coward
              Anonymous Coward

              Re: One thing that apparently happened after this story was posted

              So if Apple are not under a national security gag order what they should do is issue a statement saying that they ARE under such an order.

              They could only legally make such a statement if there was no gag order. Thus they would prove that they not in fact under such an order...

              1. doublelayer Silver badge

                Re: One thing that apparently happened after this story was posted

                While you can't be ordered to lie, you can be ordered not to disclose information. This leaves you with the following options:

                Apple: No comment.

                El Reg Readers: So clearly it's happening.

                Apple: Definitely not. We can categorically deny all of this, in any terms you like. Just read out sentences and we'll tell you that it didn't happen, to avoid any sense of our being disingenuous.

                El Reg Readers: It's almost certainly not happening.

                Judge, 2022: The government finds for the plaintiffs, owing to clear falsehoods released by the defendant in an attempt to protect them from adverse actions on their share price... [until you fall asleep]

                Apple: We can tell you that we aren't under a gag order, and that we haven't found a security device embedded in supermicro servers we purchased between the dates of ... [and other overly specific terms]

                El Reg Readers: They sound somewhat confident. Maybe we'll believe them, but we're not entirely sure.

                Meanwhile, if there really is no chip and therefore no order, you have the following options:

                Apple: No comment.

                El Reg Readers: So clearly it's happening.

                Apple: Definitely not. We can categorically deny all of this, in any terms you like. Just read out sentences and we'll tell you that it didn't happen, to avoid any sense of our being disingenuous.

                Apple attorneys: Yes, this didn't happen, but if you are that specific, someone could find a loophole and get you to say something that we could get attacked for. We don't have the time to evaluate any specific statements, so we should just issue our own denial, as specific as you think it needs to be.

                Apple: We can tell you that we aren't under a gag order, and that we haven't found a security device embedded in supermicro servers... [extra details to assure people watching that they're being honest and really trying to demonstrate that there is no cause for worry]

                El Reg Readers: They sound somewhat confident. Maybe we'll believe them, but we're not entirely sure.

                1. Anonymous Coward
                  Anonymous Coward

                  @doublelayer - effect on share price

                  What effect? Both Amazon and Apple had their share price fall the past couple days, but it doesn't appear to be related to this article, since the NASDAQ as a whole fell more than Apple did and some stocks like Netflix fell more than twice as much. You could argue "the tech industry fell over worries these attacks might be widespread" but why would anyone have that worry about Netflix? Is someone going to care that China finds out what kind of movies they like?

                  I'd argue that the story actually makes Apple and Amazon come out looking really good. They detected the attacks quickly, when they were isolated rather than widespread throughout their infrastructure, and they acted immediately to get rid of the compromised hardware. How many other companies would have even figured this out? Think about how often you read about companies that have had hackers inside their systems for months if not years undetected - and it is FAR easier to find software nasties in your systems than a tiny component the size of a pinhead on your server boards. I mean, there's a whole selection of software designed for identifying and neutralizing malware, but you're on your own finding spy hardware.

            4. ROC

              Re: One thing that apparently happened after this story was posted

              Golly - Apple doesn't maintain a warrant canary?

        2. Spazturtle Silver badge

          Re: Anonymous coward

          "Actually, we discussed the fact that the companies may have been ordered to lie."

          You can't be court ordered to lie in the US as that would violate the 1st amendment.

          1. Peter D

            Re: Anonymous coward

            "You can't be court ordered to lie in the US as that would violate the 1st amendment"

            You can be ordered to not tell the truth either directly or indirectly. Surely, sometimes that leaves no option but to lie.

            1. JohnFen

              Re: Anonymous coward

              "Surely, sometimes that leaves no option but to lie."

              I don't see how, when some variation of "no comment" is always an option.

          2. Anonymous Coward
            Anonymous Coward

            Re: Anonymous coward

            Don't the lawmakers and law enforcers treat the Constitution as toilet paper these days?

            1. mosw

              Re: Anonymous coward

              >"Don't the lawmakers and law enforcers treat the Constitution as toilet paper these days?"

              Yes, and they only use it while sending their tweets.

          3. Anonymous Coward
            Anonymous Coward

            Re: Anonymous coward

            You can't be court ordered to lie in the US as that would violate the 1st amendment.

            I doubt they'd need to be ordered to lie. If the episode is real, all companies and governments concerned will want to hush it up. All the TLAs need to do is co-ordinate and participate in the denials and offer the companies concerned a guarantee that they won't be investigated or prosecuted in relation to any such denials.

          4. simkin

            Re: Anonymous coward

            DOJ is infamous for always being able and willing to find something to indict someone for if they want. What exec is going to say no to the US government with that perpetual threat hanging over them? Not to mention national security considerations.

          5. Anonymous Coward
            Anonymous Coward

            Re: Anonymous coward

            "You can't be court ordered to lie in the US as that would violate the 1st amendment."

            "You can't *LEGALLY* be court ordered to lie in the US as that would violate the 1st amendment."

            TLAs have a certain attitude towards legal restrictions that get in the way of what they want to do, and often courts seem to indulge them.

          6. ROC

            Re: Anonymous coward

            That is not how it works with regard to the 1st Amendment. It protects speech that is for political advocacy from government interference. However, national security orders (FISA?) for businesses would not be in that category. (The other side of that coin is how Twitter, Facebook, etc can get away with suppressing Alex Jones "speech - they are private entities).

      2. Black Betty

        Re: 'None of the actors can be taken at face value

        Query regarding embedding chips in the motherboard substrate. Is this even a part of the normal manufacturing process? If it isn't, we can probably discount that part of the story as hyperbole.

        1. Rajesh Kanungo

          Re: 'None of the actors can be taken at face value

          The boards are not 'normal'. Just ask the board manufacturer to add the spy tips. In China it is not that difficult. The supply chain has been infected all the way to the component level.

          1. Bob H

            Re: 'None of the actors can be taken at face value

            Embedding things in the PCB is really, really hard and the PCBs are made in a different facility to the assembly. Firstly it would be really hard to embed something in the substrate and require a completely different process at the facility it was made in. Then It would be totally random where those boards were inserted and they would possibly have to bypass the normal quality checks on arrival. They could only be targeted if multiple people along the supply chain were involved in the conspiracy and being coordinated.

            1. ROC

              Re: 'None of the actors can be taken at face value

              The report states that there was " one version" with the PCB-layered chip, not all.

              That would make one wonder about the expertise, resources, and authority required to vary the modifications in such different ways, and why those differing techniques would be chosen among.

        2. Anonymous Coward
          Anonymous Coward

          Re: 'None of the actors can be taken at face value

          Query regarding embedding chips in the motherboard substrate. Is this even a part of the normal manufacturing process? If it isn't, we can probably discount that part of the story as hyperbole.

          That's roughly what I was thinking. Embedding a physical device, no matter how small or how smart, is such absolute proof of where the attack was carried out that it seems far too clumsy, and far too likely to be found out.

          Still, as others have pointed out, none of the actors in the story necessarily inspire one with confidence of truth, while all have something to gain from being manipulative.

          But overall, this sounds more like the current equivalent of a Red under the Bed hysteria which seems so boringly cyclic in some parts of the world, the Chinese being the bogeyman-de-jour. Where's Arthur Miller when he's needed?

        3. Sykowasp

          Re: 'None of the actors can be taken at face value

          With silicon thinning (already used for HBM stacks, for example), you could easily stick the silicon into the motherboard substrate, between standard layers. The bulge would be imperceptible, and the thin silicon might not register for x-rays or other hardware scanning solutions.

          I presume this chip is installed onto a serial data link to the flash memory, and on power on it intercepts the serial bitstream from the flash, and adds enough to install its payload.

          The hardware security solution to this is on-board flash and memory on the server management processor, preferably on the same die, made with security hardening techniques.

          1. Anonymous Coward
            Anonymous Coward

            Re: 'None of the actors can be taken at face value

            "The hardware security solution to this is on-board flash and memory on the server management processor, preferably on the same die, made with security hardening techniques."

            Thus forcing them to replace that chip, find a way to bypass it, or to corrupt it in some way.

            Interfering with the manufacturing process, with sufficient technical skill, seems to be almost unstoppable. Any 'solution' can be obviated, bypassed, or removed.

        4. Electronics'R'Us

          Re: 'None of the actors can be taken at face value

          The short answer is it can be done, but at PCB fabrication, not assembly of the entire board.

          We have been embedding small components inside PCBs for quite a while.

        5. JohnFen

          Re: 'None of the actors can be taken at face value

          "Query regarding embedding chips in the motherboard substrate. Is this even a part of the normal manufacturing process?"

          This is not a common thing to do, as it increases manufacturing costs, but it certainly has been done (for legitimate, not sneaky, reasons) before, so it's something that is a manufacturing option.

        6. Adrian 4

          Re: 'None of the actors can be taken at face value

          It's not part of the normal manufacturing system for conventional boards, though there are high-density stack-chip manufacturing methods with similarities. I think it could be done, at considerable cost and inconvenience. So unlikely in high volume, but possible for 'specials'.

        7. newspuppy

          Re: 'None of the actors can be taken at face value

          The embeding of active/passive components can be traced to the 1980's... and IBM published a paper

          ( https://www.jstage.jst.go.jp/article/jiepeng/2/1/2_1_134/_pdf/-char/en )

          <QUOTE>

          Embedding components inside a PCB motherboard or a substrate provides literally a new dimension to achieve the needs of today’s high end electronics manufacturing. Component embedding inside a substrate is not a completely new idea, and several technology approaches have been in development over the years – the first real attempt to commercialize an embedding technology was done by GE in mid 80’s.[1] But only now has the market evolved to accept component-embedding solutions and at the same time the infrastructure has matured to a level where component embedding becomes a commercially viable solution.

          </QUOTE>

          so there are no technical obstacles...

        8. Cragganmore

          Re: 'None of the actors can be taken at face value

          Like this you mean...

          https://www.electronicdesign.com/embedded/use-embedded-components-improve-pcb-performance-and-reduce-size

        9. Anonymous Coward
          Anonymous Coward

          Re: Chips in the substrate

          Yes, it can be done. Inserting resistors and diodes is common; inserting chips and capacitors I've only personally seen once. But then I worked in the software side of a defense company, so I had little exposure to this stuff. Search on the website of a company like Curtis Wright and if you can find a photograph of a processor board without metalwork, then you are likely to only see large BGA's and a few interface chips (the devices that need the cooling) and an otherwise bare PCB surface (to allow a ground plane on each side of the PCB stack to cut down on EMC.)

          Your real question should be can it be done cheaply enough, with the follow up of what is the budget for this information?

        10. Anonymous Coward
          Anonymous Coward

          Re: 'None of the actors can be taken at face value

          A well-funded operation, meaning tens of millions of dollars, would introduce pre-altered motherboards into the factory ahead of component placement.

      3. Ian Michael Gumby
        Boffin

        @AC Re: 'None of the actors can be taken at face value...

        Your personal take of a false flag by the US is laughable and not even worthy of a B movie (direct to hulu) or something like that.

        Look, Trump's beef w China and tariffs are more than just trade. He wants to apply pressure on NK.

        At the same time... you don't even think about China's activities surrounding their man made island which they now have put military forces on and are claiming ownership over some oil fields that are supposed to be owned by Viet Nam (IIRC).[Note: I could be wrong about the other countries involved... going from memory]

        I'd say that Bloomberg's reporting seems to be accurate consider ancillary factors going on.

        Do you really bork a bunch of hardware over a firmware upgrade that has malware associated with it? Or do you just upgrade to a fixed release? Or go back to a prior release?

        There's more, and what's interesting is which motherboards... blades.

      4. Wzrd1 Silver badge

        None of the actors can be taken at face value, particularly with national security implications.'

        But overall being a huge national security issue and potential shock to the US Cloud industry, its almost certain big tech would have been told to keep quiet and indemnified from any markets issues.

        And a fairly large segment of civilian US Government agencies and projects operate on AWS platforms. Such as the compromised more than once NFIP, aka NVITS NFIP Virtual Information Technology System (NVITS) (NFIP being National Flood Insurance Program). I could mention more, but at a severe cost, due to an NDA. As NFIP is going to close down, due to a lack of congressional funding, that NDA is entirely moot, as is flood insurance in the US.

    2. thames

      Re: My take?

      There seems to be a general rule of thumb that when US intelligence departments leak alarming stories via compliant press contacts, it's usually the case that the US is already doing this themselves and are sweating buckets over the thought that someone else might be doing it as well. We saw exactly this in the run-up to the Stuxnet reveal, and we saw exactly this in the backdoors being installed in Cisco networking equipment.

      I remember the same sort of vague but alarming stories claiming that foreign powers were infiltrating SCADA systems and could use that do destroy utility equipment. They even built a lab type setup of a diesel generator with attacked SCADA system and demonstrated it. Meanwhile the utility industry scratched their heads in puzzlement, because despite the alarm and panic in government, industry couldn't pry any actual details out of them so they could take preventive action and nobody was seeing it in the wild. And then the Stuxnet story came out and we found out the panic was about how the US (with the assistance of Israel) had infiltrated the SCADA systems controlling Iranian enrichment equipment and was using it to conduct sabotage and the US were afraid they would be hacked back.

      To go back to the mysterious motherboard chips, if this was real, I would expect someone to present actual hacked hardware along with demonstrations of what it did. After all, if the story were real then it's not like Chinese wouldn't already know everything about it, so what's the point of hiding it?

      And Amazon's and especially Apple's denials are pretty strong. If they were obfuscating the issue, then they would just release their usual vague waffle.

      I suspect this story is complete bullshit. The use of a security company in Ontario Canada is also very interesting. At this very moment the US is putting lots of pressure on Canada to try to get them to ban Huawei equipment from important Canadian networks. It would not be surprising if this whole story were to be an exercise intended to pressure allies into stepping into line behind the US in freezing Chinese tech companies out of western markets in favour of equipment that has the backdoors of "friendly" countries in it.

    3. Rajesh Kanungo

      Re: My take?

      Actually, Bloomberg would have been sued by Apple, AWS, Supermicro by now. Bloomberg had multiple sources confirming the insertions.

      1. Anonymous Coward
        Anonymous Coward

        Re: My take?

        "Actually, Bloomberg would have been sued by ..."

        Sued for what? Doing their job of running a free press? Nobody is doubting the sincerity of their report, whether it's mistaken or not.

  2. Anonymous Coward
    Holmes

    McLean isn't just any suburb

    Just a note - McLean, Virginia is where CIA Headquarters are located.

    1. Malcolm Weir Silver badge

      Optional

      The CIA Headquarters are in Langley, VA, which is outside McLean. MITRE is in McLean, though, as are many other organizations that one might expect to be providing support to the government. It is far more plausible that a meeting with a number of industry executives would be held off-campus than at an agency HQ, even if the host of the meeting was an agency (a meeting at a think-tank could be about anything, so the security issues are easier to handle).

  3. Grikath

    My biggest problem with Kieran's article, is that it definitely needs the Touch of an Editor, because there are three sections with Purpose Creep.

    "Cleverly" written, attributable to "condensing", but ultimately Really Bad Language.

    Given the report, there will be an army of BOFH's ripping out MB's and minutely inspecting them, so we'll know soon enough. But K. is guilty of "tendentious reporting" , to use the polite European term.

    1. diodesign (Written by Reg staff) Silver badge

      Re: Grikath

      Sounds more like you just disagree with Kieren's points. We spent pretty much all day putting it together while everyone was either rushing in to cut'n'paste Bloomberg or Apple/Amazon/SM's denials.

      Instead, here's a technical breakdown of the matter at hand, looking it from both sides.

      If you don't like it, Zzzzzdnet's that way -------------->

      C.

      1. alain williams Silver badge

        Re: Grikath

        I don't want to sound like a sycophant but I do want to congratulate El-Reg for putting together an excellent article about this -- even if, having read it, I still don't know how much truth there is in it.

        1. Peter2 Silver badge

          Re: Grikath

          Given the report, there will be an army of BOFH's ripping out MB's and minutely inspecting them

          As one of those BOFH's, i'm going to comment that there is no earthly point doing that for >99.9% of BOFH's. To make it worthwhile, you'd need:-

          1) The original plans sent to the fab.

          2) the ability to check the motherboard for objects that shouldn't be there that are on the nanometer scale.

          In addition, after you've found something that you think might not be there then you'd need:-

          3) the ability to figure out what the hell things are down to a scale of ~50nm. Xray scanners are not particularly common, and most of those aren't going to resolve down to the level where you can recognise components inside a chip, let alone allow you to identify them and spot things that have been added to the original design.

          4) the ability to pull the embedded code off microchips to figure out what they are doing is as per the design.

          Yeah, um. Next to nobody has #1, and I suspect the number of teams with the ability to pull off 2 is very, very limited. 3 & 4, um. I'm thinking "count them on your fingers".

          1. DropBear
            WTF?

            Re: Grikath

            "the ability to check the motherboard for objects that shouldn't be there that are on the nanometer scale."

            So, um, you reckon they did all that firmware hijacking via a single flip-flop...? Because double or single digit nanometer scale is what individual features of a single transistor are at, not any fucking chip of any fucking level of complexity.

            1. Anonymous Coward
              Anonymous Coward

              Re: "fucking chip"

              If you have access to a "fucking chip of any fucking level of complexity" I know of a sex toy manufacturer who would be very interested, for both straight chips and back-doored chips.

              1. Robert Helpmann??
                Pint

                Re: "fucking chip"

                ...I know of a sex toy manufacturer who would be very interested, for both straight chips and back-doored chips.

                I can only provide one up-vote, but see icon for bonus. Happy Friday!

            2. Peter2 Silver badge

              Re: Grikath

              So, um, you reckon they did all that firmware hijacking via a single flip-flop...? Because double or single digit nanometer scale is what individual features of a single transistor are at, not any fucking chip of any fucking level of complexity.

              If we are being paranoid enough to check this stuff, wouldn't we be paranoid enough to check that what is in the chips matches what we expect to be in chips?

              1. Electronics'R'Us

                Re: Grikath

                If we are being paranoid enough to check this stuff, wouldn't we be paranoid enough to check that what is in the chips matches what we expect to be in chips?

                In some industries, we do that all the time because of component counterfeiting (a major problem for kit that needs to be available for 20 or more years),

              2. Anonymous Coward
                Anonymous Coward

                Re: Grikath

                "wouldn't we be paranoid enough to check that what is in the chips matches what we expect to be in chips?"

                That's not mayonnaise...

                - because it's Friedey, of course.

          2. Stoneshop
            Devil

            3 & 4, um. I'm thinking "count them on your fingers".

            3) the ability to figure out what the hell things are down to a scale of ~50nm. Xray scanners are not particularly common, and most of those aren't going to resolve down to the level where you can recognise components inside a chip, let alone allow you to identify them and spot things that have been added to the original design.

            Well, about that 27" CRT that's still in storage for reasons unknown (except to the BOFH ... ).

          3. Nick Ryan Silver badge

            Re: Grikath

            You're not going to get much processing out of anything at ~50nm in size. In this case there is a claim that a chip has been inserted and while it's small, it's going to be nowhere near ~50nm.

            I am dubious mostly because of the speed and heat issues - in order to intercept and modify what is effectively cross bus information on the fly the intercepting chip needs to be rather fast otherwise the communication between components will be unreliable and a fast processing chip tends to generate heat. Put a generator of heat inbetween the substrates of a board and you're asking for (thermal) trouble.

            Not impossible, of course, but rather unlikely. It's much more likely that a chip is inserted just on the board itself because this is going to be somewhat easier to achieve. Or alternatively to just modify the software that is on these devices in the first place - no physical trace at all then.

  4. Dabbb

    Why bother ?

    How about the fact that most commonly used server management software and hardware formerly known as Emulex Pilot now owned by Chinese Aspeed ?

  5. Anonymous Coward
    Big Brother

    Chinese agents slip spy chips into Super Micro servers

    Wouldn't it be simpler to activate the Intel ME backdoor, the backdoor that the end user can't disable except for the NSA? The backdoor that Intel forgot to lock-down with a password. Remember Trusted Computing doesn't mean you can trust your computer what it really means is the spooks can be trusted to have a backdoor into it.

    1. Flocke Kroes Silver badge

      Words mean different things to different people

      In the security services, "trusted" means "someone who can betray you".

    2. Anonymous Coward
      Anonymous Coward

      Re: Chinese agents slip spy chips into Super Micro servers

      The ME backdoor requires you already have access to the local LAN to exploit it. This spy chip attack is can be leveraged from halfway around the world, it would only fail if the network the server is on is completely isolated from the internet.

      Plus the ME backdoor goes away once it is found and patched. The spy chip attack lives for the life of the hardware, with no way to disable it short of putting the motherboard in an industrial shredder.

      1. Ken Hagan Gold badge

        Re: Chinese agents slip spy chips into Super Micro servers

        "The ME backdoor requires you already have access to the local LAN to exploit it."

        How much access would you need, though? The ability to send a particular network packet might be sufficient to let you exploit the ME in a machine next door to you and once you have better-than-root privileges on one machine it probably isn't hard to work your way around the whole LAN and out to the internet. So ... you start by sending dodgy emails to non-technical staff.

    3. Rajesh Kanungo

      Re: Chinese agents slip spy chips into Super Micro servers

      The Intel ME bugs may not have been known or there could have been the fear of the bugs being fixed. Moreover, any attacks would take known paths so they could be blocked.

    4. phuzz Silver badge

      Re: Chinese agents slip spy chips into Super Micro servers

      The report says that the component was designed to hack the BMC, which can already do everything that the IntelME can do and more. So yes, using a vulnerability in ME might well be easier than surreptitiously inserting a component, but it's not like most BMC's are any more secure, and would probably be a much easier target.

      (They also don't mention what kind of CPU these boards had. They might have used AMD or even ARM CPUs, although given how many Intel based servers there are out there, it's unlikely)

      Another possibility if you had some access to the boards during manufacturing would be to just swap the BIOS (or BMC, or a number of other chips) with one that contained some kind of malicious capability.

      Basically, there's easier ways to do what is being claimed, and attacking IntelME is just one of them.

      1. doublelayer Silver badge

        Re: Chinese agents slip spy chips into Super Micro servers

        There are easier ways to have a backdoor, but this way is pretty good for having a backdoor that's hard to spot. If you simply replaced the chip containing the BIOS, made a backdoored flash chip, etc. then all you'd need to do to find them is to test that chip, as they do just to make sure they're working. If, for example, you took a flash chip and asked for its contents, it would be instantly obvious whether the contents were right or not. By having a separate chip to handle that, you would have to test all components of the board together, and that only helps if you know what to be looking for. For the people doing this, it would actually be easier just to see if you can find the chip in the board. So I don't know whether this chip was ever created or installed, but the details make sense if it was.

        1. Richard 12 Silver badge

          Re: Chinese agents slip spy chips into Super Micro servers

          Why embed into the motherboard substrate? That's really expensive and subject to failure.

          If I were doing this and had that piece of silicon, I'd embed it into the packaging of a chip that's supposed to be there.

          Cheaper and more reliable as the chip packaging is designed to do this. At least as hard to detect, possibly more so as multi-die packaging is very common.

          Of course, the simplest way to do this kind of thing is to swap out the content of a flash chip.

          If this attack is real, I am pretty sure that there was no custom silicon involved whatsoever, it will be a firmware image attack as that's cheaper and harder to detect as there are no visible indicators at all.

          1. Stoneshop
            Facepalm

            Re: Chinese agents slip spy chips into Super Micro servers

            Why embed into the motherboard substrate? That's really expensive

            And this would be an issue for the Chinese entities purportedly involved, exactly how?

      2. Stoneshop

        Re: Chinese agents slip spy chips into Super Micro servers

        (They also don't mention what kind of CPU these boards had. They might have used AMD or even ARM CPUs, although given how many Intel based servers there are out there, it's unlikely)

        SuperMicro (as the suspected manufacturer) has just a small number of AMD boards in their (extensive) product range, and exactly zero ARM boards.

  6. TReko

    Superb reporting and analysis, Register!

    great technical and legal analysis. The parsing of Apple's and Amazons press statements is an education, too!

    1. I ain't Spartacus Gold badge
      Black Helicopters

      Re: Superb reporting and analysis, Register!

      Ah, but that is just what El Reg want you to think?

      In reality it is them that control the Chinese government, from their space station Vulture 1. Why do you think all the reporting of their rocket plane went quiet, with that flimsy excuse about the FAA not giving them a license?

      Also they have embedded cameras and microphones in all the Playmobil figures around the world. Thus every world leader with children is a potential security risk.

      Keep your tinfoil hats handy! Arm and prepare for the Vulturepocalypse! They are coming to get us a...

      ...

      ...

      ...

      ...

      1. I ain't Spartacus Gold badge

        Re: Superb reporting and analysis, Register!

        I'm sorry. You can ignore that previous post. The Register are of course the most trustworthy of sources. As well as being brilliant, sexy and very generous with buying their readers beer at their lectures.

        Trust The Register. The Register is your friend. The Register wants you to be happy.

        1. Waseem Alkurdi
          Pirate

          Re: Superb reporting and analysis, Register!

          I think I've just heard a robo-vulture's evil laugh being emitted from the stratosphere.

  7. John Savard

    Still Reason to Worry

    Oughtright lies from the companies involved would be unprecedented, whereas the Bloomberg reporters believing someone who was mistaken that SuperMicro was the unnamed target is highly plausible. But that would mean it did happen, just to someone else we don't know about.

    1. Steve Chalmers

      Re: Still Reason to Worry

      Methinks the only time it makes sense to embed a chip would be if the server were destined for a classified facility which would wipe and reload (from trusted binaries) every single byte of code on the motherboard.

      The hardware strategy would then allow the board to be re-hijacked after it was thought to have been wiped and reloaded.

      We may be hearing true story #1 about what happened, and true story #2 about where something else like simple substitution of code for a management processor occurred, but the two stories are mashed up to signal to the perpetrators that the attack is known without disclosing to anyone else where the attack actually occurred.

      Now if the perpetrator could only control the motherboard model supplied in a bulk order to SuperMicro, and only some of those boards went to my hypothetical classified site, then many other such boards could have gone to many other customers, either sitting silent or making mischief, which could be the source of a true but irrelevant statement on the number of end customers who got hardware compromised boards.

      Just thinking and speculating, no inside knowledge (and no clearance any time in my life) here.

  8. Anonymous Coward
    Anonymous Coward

    Well done El Reg Article

    Except I'm too old to see the pencil tip chip that is supposingly circled in front for us to see.

  9. martinusher Silver badge

    Let's not go overboard with this.

    Unfortunately the conduit for this information appears to be not very technical, we're told vague things about the part that would be worthy of a modern day spy thriller but don't make an awful lot of sense to someone who actually understands these designs. As its been in the boards for a decade or so we have to assume that with its form factor its not going to be anything much more sophisticated than a medium sized EEROM. It could patch code on the fly but that's more theoretical than realistic because there's no guarantee that the code its patching will be stable for an extended period of time.

    I'm prepared to dismiss this as disinformation put around by our own intelligence services (who would probably love a capability like this but they really have one already in the form of the Management Engine. I'm also used to seeing Bloomberg being used as a conduit for this sort of information -- we normally think of them as a financial site but for a long time now if you wanted a story about Russia or China planted in the media they seem to be one of the 'go to' publications.

    What is particularly worrying about this is that the overall picture I'm getting these days of our technical capability is that we seem to be losing it. I'm seeing more marketing and less technology, stories about wonder weapons, mystery capabilities of real and imagined enemies, all dark paranoia and no real technology. This dovetails rather nicely with my perceptions of industry -- obviously the picture's not all bad but in general there seems to be a dumbing down as skilled people age out and are not replaced (or replaced by people with a very different set of skills). This may end up being the story behind the story; its already old news in the UK but the US....

    1. eldakka

      Re: Let's not go overboard with this.

      As its been in the boards for a decade or so

      Sure, if 3 years (2018-2015) fits into the "or so" part of "a decade".

    2. OldCrow
      Holmes

      Re: Let's not go overboard with this.

      Let's arm-chair-design a recreation of this exploit, and see how close we get to the real thing, after all the facts come out, shall we:

      1. Since the BIOS/UEFI is still loaded from an SPI FLASH chip, which is in a very standard form-factor (read: wastefully large blob of plastic around a tiny FLASH chip), it's easy to make an identical package that houses 2 memory areas.

      Switch the memory areas after 100 hours of power-on, or after 20 BIOS-loads. Now you have control over the BIOS boot sequence AFTER the board has been tested and installed in location.

      2. Next, let's make an USB flash drive, but package it like a USB over-voltage-protector diode package. One of those small ICs that you see hugging the USB bus near the connector, in any properly designed circuit board, protecting the other ICs from your static-electricity-laden fingers.

      It'll be the largest over-voltage protector you've ever seen, but it'll still pass inspection.

      TVS diodes come in many packages. A government-standard suppressor package may be larger.

      Again, activate after 20 power cycles, if (and only if) there is no other device attached to the USB bus.

      3. Leverage one of the well-documented standard ways to do a Superfish on the Windows installation.

      4. Profit.

      Edit:

      Scratch that. Just do a proper Superfish after switching the SPI chip memory areas. No need for the USB drive after all. Left as-is for posterity.

      1. Brian Miller

        Re: Let's not go overboard with this.

        Let's arm-chair-design a recreation of this exploit, and see how close we get to the real thing, after all the facts come out, shall we:

        I read the original Bloomberg article. The way the article was written, it sounded like the "signal conditioner" chip could connect to the network, by itself! Only later on did it go into "detail" about it modifying the code for the BMC.

        What all of this points out is something very important in system design: the CPU should not boot code that it can't verify through a chain of trust. There are a number of commercially available solutions for this, and they have been on the market for years. The concepts have been out there for far longer. Manufacturers have no reason to not pursue secure operation.

        The real problem with all of this is the motherboard design has to be modified! If a shared serial bus was modified, then that means that that there will be a signals conflict on the bus to modify instructions. The problem with this is that the commands are like, "Hey, #24, talk to me!" Then #24 talks, and does it blindly. To actually do what the article claims, the chip has to be in series between the CPU and the memory. That would take a change in the traces, etc. So the motherboard would have to be redesigned to incorporate the chip.

        Whatever is going on, we aren't getting the full story yet.

        1. OldCrow
          Meh

          Re: Let's not go overboard with this.

          we aren't getting the full story yet

          Uhh... yes, we are. Getting the full story, that is. The full story is: "China switched out a memory chip. Did a delayed BIOS driver-switch attack." (Vector is actually named Microsoft Windows Platform Binary Table, not Superfish, which is a separate piece of malware. My apologies for mixing them up.)

          You can get full control of the server just by making it load an infected driver before OS boot, via e.g. UEFI option-ROM.

  10. JeffyPoooh
    Pint

    Deja vu

    Wasn't there a very similar story several years ago?

    1. blondie101

      Re: Deja vu

      Yes there was. Something to do with Snowdon and facilities of the NSA and the likes implanting spying chips on demand in HP gear. The HP designs deliberately facilitated the spy chips. Another spy agency, likewise operations.

    2. JeffyPoooh
      Pint

      Re: Deja vu

      There's also the old 'Reprogram The Embedded ARM Chips' ploy.

      Also applies to any programmable logic arrays.

      It's precisely hopeless. Best to resort to an Art of War approach, with honeypot data, fake data, and so on.

      So who is the downvoter? Seems a bit silly...

  11. Tim99 Silver badge
    Alert

    Re: Chinese Super Micro super spy-chip...

    "You know: the country that makes everyone's iPhones phones electronics". FTFY

  12. Anonymous Coward
    Anonymous Coward

    Which Nation State

    could ever possibly mandate that all PC's ship must with a backdoor in the hardware soldered onto their motherboard ?

    1. Paul Crawford Silver badge

      Re: Which Nation State

      All of them.

      But not all of them can actually deliver on that...

    2. Pascal Monett Silver badge

      Re: Which Nation State

      The same ones that argue that encryption should come with backdoors ?

  13. Jove Bronze badge

    Denials

    I suggest you read the statements again - they are not what you have concluded.

  14. Anonymous Coward
    Anonymous Coward

    "Intellectual property theft"

    Sigh...

  15. Anonymous Coward
    Anonymous Coward

    If it's true then someone must have access to a compromised board, show everyone the firewall logs, what further proof would you need? It's the on-board networking that has me suspicious, you don't need it once the system is compromised as far as I am aware and it would only serve as an extra detectable layer.

  16. TReko
    Happy

    Fun and games

    you left of some juicy details from the Bloomberg article:

    "In 2009 the company announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, a deal that paved the way for Elemental servers to be used in national security missions across the U.S. government.

    Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

    1. Michael Wojcik Silver badge

      Re: Fun and games

      Bloomberg is being overly picky about what constitutes a "sermon" and a "congregation".

      Certainly many sects observe the practice of laying on hands. And you can't fault their faithful devotion.

  17. Anonymous Coward
    Anonymous Coward

    Bloomberg is hanging way out in the breeze

    Their reputation is really hinging on this... either it's their Woodward/Bernstein "Watergate" moment, or years from now we'll be going "hahaha! remember when that Bloomberg rag spouted that story about super tiny Chinese spy chips?"

    If this isn't true, the egg on their faces will be enormous.

    1. I ain't Spartacus Gold badge

      Re: Bloomberg is hanging way out in the breeze

      To be fair to Bloomberg, it's entirely plausible. So as long as they eventually admit they screwed up - and ideally tell us how it happened - they can mostly repair their reputation.

      After all, the Sunday Times (and others) survived the Hitler Diaries fiasco. Although people still laugh at them occasionally...

  18. WatAWorld

    It is a matter of choice

    I see this as a a matter of choice.

    Your choices are to be spied on in similar ways by the Americans, the Chinese, or by one of the other big players.

    Not being spied on at all is not an option.

    1. eldakka

      Re: It is a matter of choice

      Your choices are to be spied on in similar ways by the Americans, the Chinese, or by one of the other big players.

      I'd rather be spied on by the foreigners. They are less likely to just wander into my home and arrest me and throw me into jail on trumped up charges.

      1. theModge

        Re: It is a matter of choice

        I'd rather be spied on by the foreigners. They are less likely to just wander into my home and arrest me and throw me into jail on trumped up charges.

        That would be my preference too.

        I however have a far stronger defence: realistically there's absolutely sod-all of interest on my computer. I do research, but unless more effective data integration for your trains is a matter of national security I'm quite safe. Granted cyber attacks on national infrastructure are a thing, but my work would not be a good jumping off point for that.

        1. Anonymous Coward
          Anonymous Coward

          Re: It is a matter of choice

          "here's absolutely sod-all of interest on my computer"

          I guarantee there's enough on your PC to steal your identity, lock stock and barrel.

          You might not be quite so complacent if that happened.

          1. theModge

            Re: It is a matter of choice

            I guarantee there's enough on your PC to steal your identity, lock stock and barrel.

            Of course, but whilst I'm not secure against nation state level actors who can e.g. get into the supply chain for my motherboard (who is?) I do take the sort of everyday precautions necessary to exist in this day and age, to the extent that it's possible to be secure whilst relying on the million different 3rd parties that comprise a modern computing environment.

    2. ShadowDragon8685

      Re: It is a matter of choice

      I've got an idea.

      If "not being spied upon at all" is not an option, let's go the other route - exactly the other route.

      EVERYONE is being spied on by EVERYONE.

      No; I don't just mean letter agencies. I mean Bob from Accounting. Not your Bob from Accounting; somebody else's Bob from Accounting. And not as part of his accounting duties, just what he does on the weekend for fun, he spies on the NSA.

      Give everyone - literally everyone - full and open access to the data of everyone - literally everyone. At least then the process would be well and truly fair.

  19. Steve Davies 3 Silver badge
    Alien

    It is interesting in several ways

    firstly all these MoBo's were taken out of use or never got into front line service several years ago yet the Apple and Amazon stock price dropped.

    Their current results won't be affected by this so why?

    Bloomberg has been running a lot of attacks on other tech companies this year. Their principle target has been Tesla.

    Now for the conspiracy theory.

    APPL and TSLA are two of the most heavily shorted stocks on Wall St.

    Shorters bet on stock falls.

    So the shorters buy options on the target stock(s)

    Their golfing/sailing/frat buddies release a story about something that happened years ago

    The target stocks drop

    The shorters sell the options before the end of the settlement period.

    The shorters make money, sorry, make that LOADSAMONEY.

    If only life was that simple eh?

    1. Anonymous Coward
      Anonymous Coward

      Re: It is interesting in several ways

      You're assuming the stock price drop had anything to do with this news. Google, Netflix and Tesla had a drop double the size of Apple and Amazon's and they weren't named in the story. Who would really care if the Chinese were spying on Netflix, they gonna find out about your weekend binge watching habits? Hard to see how Netflix's drop could have anything to do with this story.

      This was just an across the board tech stock drop, just like some days there's an across the board tech stock gain. Now one can argue the reason everything dropped was concern over this hack, but if so it didn't hurt Apple and Amazon worse than other companies. Indeed, Apple fell slightly less than the NASDAQ index as a whole.

      If this was an evil stock market short plot, it wasn't very well executed. You'd much rather get one or a few really big stock drops, not a minor 2-3% across the board drop. The SEC polices these things pretty well too - no doubt they will be examining the trading patterns around Supermicro stock (the only one that really took a big plunge) to see if someone sold a bunch of it short recently, or made unusual put option purchases.

      1. Anonymous Coward
        Anonymous Coward

        Re: It is interesting in several ways

        Which means that the release of the Blumberg story was not the cause of the price drop as discussed in the article. For the reasons given above, I agree. Blumberg's coverage of tech hasn't been friendly, but that's not surprising given the antics of tech "leaders" like Musk. There's a lot of BS backing up in those tech company comms lines, so it would be hard to imagine how anyone outside of Silicon Valley PR could ignore it.

        As for the substance of this report: I'm with the many above who think it's part of a deliberate disinformation campaign to bolster the trade war against China while providing cover for a non-Chinese operation to hack cloud services.

        The bottom line is you can't believe anyone any more. Those in the know are either incentivized to lie for various reasons, or gagged by like local secrets laws. That could all change if the press and public demanded actual hard evidence when faced with these kinds of claims. But that's unlikely because the press is complicit and the public disorganized. Now if media outlets were to experience sharp, sustained drops in _their_ income due to the public's ceasing to trust them...

        1. Anonymous Coward
          Anonymous Coward

          Re: It is interesting in several ways

          What hard evidence could they POSSIBLY provide that would change the minds of those who believe this is a US government plot against China? If they made samples of the hardware available for people to look at, how can you tell it was designed by China instead of the US? If they had logs of the chips contacting a Chinese controlled C&C server, how do you know that really happened, or that C&C server wasn't a CIA front? Hell, if they had emails from China's president to their hackers saying "have we stolen Apple's A12 design yet?" signed with his private key, they'd claim the NSA has the technology to break the email encryption/signing that China's president uses.

          Once people go down the conspiracy theory rathole, no amount of evidence can possibly change their minds. Everything you show them will become part of the conspiracy. Look at the moon landing deniers, who have crazy explanations for everything from lunar dust samples to the retroreflective mirrors astronauts left behind that are part of the grand conspiracy to fool people into thinking we landed on the moon.

  20. Tom 64
    Windows

    From Amazon's denial

    > "It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China".

    Umm, what about outside China then? Like in the USA?

    Breaking out the popcorn. The fallout from this one will be entertaining.

    1. Anonymous Coward
      Anonymous Coward

      Re: From Amazon's denial

      The story said Amazon had only found servers with the spy chips in their Chinese datacenter. Apple found it in a limited fashion as well. These weren't across the board "wow, we have to replace everything in all our datacenters".

      It seems both companies do some very robust checking of the boards they are shipped, since it was noticed rather quickly. It goes without saying that 99.99% of companies don't even check the boards in their servers to look for a tiny component that doesn't belong. The main reason Amazon and Apple did was because like other major 'cloud' companies such as Google, Facebook, Microsoft etc. they design their own boards, and want to verify what they are shipped matches their design.

      That's why the spy chip was apparently disguised to look like a passive component, hoping it would be ignored as something to address RF or electrical issues.

    2. Rajesh Kanungo

      Re: From Amazon's denial

      The first case was from the acquisition. So yes, Amazon probably knew about it happening on US soil.

  21. Muppet Boss

    "At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems."

    I assume this means that modified hardware or malicious chips were working just fine? No firmware issues like Apple's?

  22. MrBlack

    'Bloomberg Terminals, with many layers of editing and fact checking, and it has a zero tolerance on errors: it is inconceivable that it would publish a story this huge that wasn't watertight.'

    A bit of a bold statement, no ?

    1. Anonymous Coward
      Anonymous Coward

      Not bold, simply logical

      Bloomberg charges a LOT of money to traders for their information, including early access to their stories. They aren't going to risk a multi billion dollar business on a story they aren't 100% confident in. That doesn't mean they can't get fooled if the CIA threw its full weight behind trying to fool them, but I think it is more likely that Apple and Amazon's denials are either wrong or were somehow coerced.

      Most likely wrong - if only a few people in each company learned of the issue before contacting the FBI, and they were then told not to tell anyone else, yesterday's denials are easily explained. Press contacts Apple / Amazon spokesperson for comment. They contact various executives asking "do we have any comment on this story" and the executives all know nothing about it and neither do their underlings they talk to - because the odds of finding the three or four people who do know about it are tiny - so the spokesperson reports back that the story is false.

      The employees who apparently went to the press (perhaps they were worried other US companies without their resources would be unknowing victims if the government kept the story hush hush forever) aren't likely to fess up to others in their company now - because it would be obvious they had leaked info to the press about it and the company would think "if you leak this, you might leak other stuff, so here's your severance package goodbye"

    2. I ain't Spartacus Gold badge

      it is inconceivable that it would publish a story this huge that wasn't watertight.'

      A bit of a bold statement, no ?

      MrBlack,

      I suspect "watertight" here is journalese. You can't guarantee that every story you publish is correct. Even historians with the benefit of multiple sources and hindsight can't do that. What you can do is check that your story stands up and is watertight. So you use multiple sources and do some checking on them, to make sure they've not got obvious motives to lie - or links between them that suggest a conspiracy. Once you've done all that you go off to the lawyers and make sure that your methods will stand up to getting sued - i.e. you can say you did all possible checking.

      Often the lawyers will then send you back to do more homework. With something like this you'll obviously have to satisfy the editor, but also upper management, on the grounds of the financial risk to the company.

      After all that, you then approach the people the story is about to see if they're willling to respond. And then do any more checking on what they say. If they deny everything then you have to meet an even higher standard of proof internally, because the legal risk has just shot up massively.

      At some point you've then done all you can do - and then have to decide whether to publish or not. If the victims are denying everything, then you either trust your sources and research and risk it - or do nothing. If you do nothing, you may never find out the truth. If you pubilsh, there's a much higher chance the truth will come out. But obviously a much higher risk of getting sued, or being made to look really stupid.

  23. Scott Broukell

    Should we be worried ?

    We recently had our pet cat `chipped' by a vet of Chinese ethnicity and she now spends increasing amounts of time curled up right next to the broadband router!

    1. Anonymous Coward
      Anonymous Coward

      Re: Should we be worried ?

      You should be worried, your cat is being affected by wifi radiation and will soon get superpowers like being able to leap 5x its height and fit itself into even the smallest box you have left lying around. If your cat already can do these things, it is too late....run!

      1. Doctor Syntax Silver badge

        Re: Should we be worried ?

        "and fit itself into even the smallest box you have left lying around."

        But you won't know if it's still alive until you open the box.

    2. nagyeger

      Re: Should we be worried ?

      This sounds like a great ignoble prize research topic: (chipped) animal behaviour influenced by external RF sources, via the nice warm neck (or wherever the chip got put) syndrome. What you need to do is set up another (identical) router a few feet away and alternate which one has it's WiFI transmitter on. Correlate with cat's favourite resting place...

    3. Stoneshop
      Boffin

      Re: Should we be worried ?

      Have you had shipments of tuna, salmon and catnip arrive at your door even though you're completely sure your cat has not touched any of your computers? As those implanted chips are small and low power they need to be really close to the access point to be able to connect, that's why you will find your cat right next to it.

  24. Giovani Tapini

    Only just passes the plausibility test for me...

    I agree it is completely feasible that a board maker could have it's design compromised. However if you look at a motherboard it is not straightforward to add any components at all. There is not a lot of room, chip embedded in the fiberglass is likely to overheat running at CPU speeds and burn itself away, getting access to the right tracks will be non trivial. It would probably also upset the fine balance of power management that modern motherboards don't have much tolerance for.

    Overall if I am to believe the theory I would expect this to have been achieved by the motherboard designers, not just bodged in at the factory. This idea that its factory changed I find completely implausible.

    If you are doing this in the highly managed environment of an AWS (for example) datacentre, the network traffic is so highly managed it seems unlikely that even if data is capable of being siphoned, your ability to trigger the siphon and retrieve anything is highly unlikely to be successful. It almost certainly would be unable to be contacted directly even with insider help. This leaves the possibility that it is trained to "look" for certain data streams to activate. Again how to exfiltrate, particularly if done in bulk across a whole datacentre, I can only imagine it would have to insert the data into what appears to be legitimate traffic, a sort of steganography. Trying to get anything coherent out a a vast number of servers operating in parallel seems both impossibly hard, and highly likely to be detected.

    Overall my assessment is that this is likely to have been rasied as a potential attack vector, has been validated by the various anonymous sources, but likely has never been attempted at scales as described.

    1. doublelayer Silver badge

      Re: Only just passes the plausibility test for me...

      You are right about a lot of this, but have missed a few points:

      It would indeed burn itself out and use too much power if running at CPU speeds. It doesn't need to. If the story is correct, it only needs enough processing to inject code into a serial line. That takes a lot less power. After this, the CPU handling the BMC handles all the work.

      It probably wasn't (if it exists) created by the factory. Instead, the plans would have been created elsewhere, and a slight modification to the process would be necessary. I don't know much about the organization of Chinese motherboard factories, but if I had plans that were almost identical, I assume the factory could build them just as well.

      The point about monitoring internet traffic is a good one. I don't have a great explanation for how that worked. The best I can come up with is that you could set up an image on such a system that could interact with the firmware and exfiltrate information into that VM, then hide the data as it is sent out from that VM with other expected traffic. Still, that's hard. If it actually exists and was used (it could be a sleeper system for some purpose), perhaps some network traffic systems aren't as thorough as we hope.

    2. Stoneshop

      Re: Only just passes the plausibility test for me...

      If you are doing this in the highly managed environment of an AWS (for example) datacentre, the network traffic is so highly managed

      Piggybacking via steganography on entirely legitimate data connections to an AWS cloud.

  25. Archivist

    Signal conditioning chips

    Aren't chips (ICs) as we normally describe them but passive surface mount filters. It is extremely unlikely that power is available on any pads they are soldiered to.

    The whole thing is bollocks.

    1. onefang

      Re: Signal conditioning chips

      "It is extremely unlikely that power is available on any pads they are soldiered to."

      There is this thing called phantom power. You use the data signal for power. I know this sort of thing was being done in the early '90s, the company I was working for did it for MIDI devices. Could suck enough juice from the MIDI signal lines to power the microcontroller and the rest of the circuitry, no need to plug in a wall wart.

      1. MacroRodent

        Re: Signal conditioning chips

        Also, these would not necessarily have been placed where a filter would have been, but somewhere with a +5v line nearby, for example. Who says the motherboard design was not slightly altered to accomodate them? Only an expert familiar with the non-tampered layout would notice it.

        1. Giovani Tapini

          Re: Signal conditioning chips

          Would take an expert to make the design change too, I would expect tampering like this to be invisible from a visual inspection, and only potentially possible from a photographic comparison particularly if it is intentionally obscured in the layers, or using another surface mount chip as cover.

      2. Anonymous Coward
        Anonymous Coward

        Re: Signal conditioning chips

        Yep. It got stolen by the computer industry and is now called Power Over Ethernet. It isn't actually using data signal for power, it just uses the principle that you can run DC and AC on the same cable and separate the two at the other end with a capacitor. POE uses separate cores but it uses the same current driven technique. I love it when rock and roll infects the rest of the world !

        Either way, power is power. As long as you what you suck is less than what will cause the receiving end to fall over, you are good to go.

  26. Anonymous Coward
    Anonymous Coward

    In an era of trade wars and when demonising China is a major populist dog whistle this may have another layer of onion skin to unpeel.

    One might be labelled a conspiracy theorist to voice it but a story, true or not, that ties Chinese government hacking and off shore manufacturing by an American company in with a major security breach could be spun as a powerful warning to on-shore your manufacturing facilities and to "buy 'murrican".

    2% off Apple's share price isn't just loose change down the back of the sofa, it will have made people panic and the 'drip drip' of anti Chinese manufacturing news could eventually force a rethink by some major players which could have severe implications for the Chinese economy.

    1. Steve Davies 3 Silver badge

      The MAGA angle

      2% off Apple's share price isn't just loose change down the back of the sofa, it will have made people panic and the 'drip drip' of anti Chinese manufacturing news could eventually force a rethink by some major players which could have severe implications for the Chinese economy.

      Which is exactly what El Trumpo wants is it not?

      Quite where they are going to find all the workers needed when say Apple brings their iDevice production to the USA is another matter entirely. Perhaps all those Coal Miners who have been led up the garden path by Trump and his 'Good Clean Coal' promise when mines and coal fired power plants have been closing at a faster rate than before he got the nuclear button codes.

    2. ROC

      MAGA angle indeed!

      With all the known Chinese cyber attacks (siphoning data in all imaginable ways - IoT, routers, cell phone apps, etc), IP espionage, South China Sea aggrandizement, internal persecution, and external persecution of the likes of the Dalai Lama, they manage to demonize themselves quite well without external help.

      If this is some "plot" to get more critical electronics manufacturing (consumer would be nice, too...) moved from their jurisdiction to about anywhere else in the world (not Russia of course, if they could even do it...), then I am all for it.

  27. Rol

    The idea that these companies denials could be later banked by a gaggle of lawyers acting on behalf of investors is the only argument in favour of believing their statements to be true.

    On the other hand, their defence would no doubt revolve around the fact American intelligence services more or less written those statements for them.

    1. Roj Blake Silver badge

      Executives of large companies think short-term, not long. Shoring up the share price today is more important than preventing a collapse in a few years time when they will have cashed in on their stock options and moved on.

      Hence the denials.

  28. Anonymous Coward
    Anonymous Coward

    It's simpler

    "A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacturer, using bribes and pressure?"

    As in the subject. It's simpler to add a chip rather than have a secondary, backdoored, CPU line.

    1. Rajesh Kanungo

      Re: It's simpler

      Agreed. Also, adding extra circuitry in the larger processor is probably easier to catch during root-cause analysis than something you never knew existed.

      1. Anonymous Coward
        Anonymous Coward

        Re: It's simpler

        "As in the subject. It's simpler to add a chip rather than have a secondary, backdoored, CPU line."

        "Agreed. Also, adding extra circuitry in the larger processor"

        It's unlikely they're talking about a different processor, merely a different firmware for the same processor, so it's just a matter of reflashing something, which is technically much easier.

    2. Steve Chalmers

      Re: It's simpler

      The only case where this makes sense is if the attacker knows the server is going into a classified environment where as a matter of course every single bit of code on the motherboard will be wiped and reloaded before the server is installed. The hardware technique would allow the server to be re compromised after it was thought to have been scrubbed.

  29. Rajesh Kanungo

    Entirely plausible

    Completely plausible. The best proof would be an actual Supermicro board with the spy chip. My only question is, why do it and get a bad rap? My bet is that China has multiple approaches and this got caught. Moreover, China may have realized that the US was aware of the spy chip. So the US went public with it.

  30. hk

    Cui bono

    I really wonder what's to gain from industrial espionage on companies like Amazon or Apple. The surely are advanced in operating their data centers with advanced technology, custom kernels and partially silicon. But that's still a far cry from some super secret formula.

    1. SImon Hobson Bronze badge

      Re: Cui bono

      I really wonder what's to gain from industrial espionage on companies like Amazon or Apple

      It's hinted at in the article - its not Apple or AWS they are targeting, but the end users of those systems. Compromising the manufacturing of their systems means you can get compromised machines into places that would otherwise be hard to get compromises into - and thus give you another attack vector into some "quite well defended" territory.

      Also, as to the "why not just adapt another chip". Well if the manufacturer sticks a JTAG clip onto the flash ROM to put new firmware into it, your separate chip can sit there all safe and sound - and un-noticed. And don't forget that if true, this was done by people with access to the skills and technology to make it happen - it's not like you or me "hacking" a built board, it's the people who make the boards using a slightly modified design. A chip buried in the layers would be invisible, and if buried underneath an existing chip would even be (more or less) invisible to x-rays.

    2. Anonymous Coward
      Anonymous Coward

      Re: Cui bono

      Azure? Google Cloud?

  31. Anonymous Coward
    Anonymous Coward

    Seems like Rube Goldberg approach to spying

    There are tons of legit chips on the board. Instead of adding one which will be certainly discovered sooner or later, why not use one of the chips that are already there?

    Just fake a vulnerbility in one of the legit chips, and if discovered, then it is just a bug, like so many others. Make it a hardware bug like Spectre or a well hidden firmware bug.

    To me, adding an extra chip to do spy is a Rube Goldeberg solution to spying. The same goal could be achieved easier, with less chance of discovery and with better deniability when discovered. Just don't fix one of the vulnerbilities discovered during development of the board. Maybe even hide it a bit better. And if discovered: no biggy, because bugs happen - to everyone.

    1. doublelayer Silver badge

      Re: Seems like Rube Goldberg approach to spying

      That doesn't work. If anything modifies those chips later (I.E. the manufacturer updates something), your bug is destroyed. If the chip is tested, you are discovered. And you can't easily make new holes in the thing because you didn't design it. With a separate chip, the manufacturer updating a chip can't kill you, a test of a component cannot find you out, and you can use all those existing chips to hide yours, which can be really tiny and be set under another chip.

      1. Malcolm Weir Silver badge

        Re: Seems like Rube Goldberg approach to spying

        Not so: if (as another poster has described) you have an SPI memory with a "secret" bank and a regular one, and the sneak chip flips between the two, why would you not also have it treat the JTAG interface in a similar sneaky fashion (i.e. write the new code to another bank)?

        Remember: the implication is that these are custom designed parts to do the job, not commodity parts being used to carry malware.

        1. Anonymous Coward
          Anonymous Coward

          Re: Seems like Rube Goldberg approach to spying

          It risks being spotted in a detailed x-ray, especially if the target is state-trusted agents who will be wary of such trickery.

          In which case, your only bet is something so oblivious-looking that it gets overlooked.

  32. Potemkine! Silver badge

    A component inside PCB layers...

    ... would be possible to implement but would have to be thermally very innovative: an active device like this one dissipating power into a sandwich of low-conductive fiber-glass panel would require some copper nearby or at least vias to evacuate the heat, making it more visible.

    1. DropBear

      Re: A component inside PCB layers...

      I'm not so sure. I don't think the chip does very much most of the time on average, excepting the immediate bus-hugging logic I certainly wouldn't expect it to run its internal processor (if it even has one) at anything like "CPU" level speeds. Besides, all it really needs anyway is a good connection to the internal ground plane - using that for thermal dissipation at least partially instead of a conventional heat sink is already standard practice for certain types of designs...

      1. Ommerson

        Re: A component inside PCB layers...

        Multi-layer PCBs have power and ground planes - entire layers that are pretty much entirely metalled with copper. Quite some heat capacity there. This answers the question elsewhere about where the power comes from - directly underneath through a via.

        In fact, embedding a device into the middle layers of a PCB is genius - likely to evade optical inspection.

  33. Anonymous Coward
    Anonymous Coward

    post-truth times

    there's absolutely no way to come even _close_ to the actual truth, i.e. what happened, and whether it actually happened. There's simply too much at stake for all parties (allegedly) involved - billions of USD combined with "national security" (and red buttons) and, of course, the battle is the world top dog. So, the best we can expect is a blockbuster movie (I bet the screenplay's already half-completed). Sad times.

  34. Jamie Jones Silver badge
    Joke

    "Apple's denial is typical Apple. Reflecting its superiority complex"

    There goes the big exclusive apple were going to give you (!)

  35. Bibbit

    Good article

    Thanks for breaking it down. I had no idea Bloomberg hacks are renumerated due to market moves either.

  36. iron Silver badge

    Having some experience of chip and pcb design and having worked with contract manufacturers I highly doubt Bloomberg's story. Sure its not impossible but it would be a damn site easier to compromise the firmware, IME or just use a flaw like Spectre or Meltdown.

  37. Anonymous Coward
    Anonymous Coward

    Second chip added, bollocks

    This is a nation state with silicon foundries.

    Why would you add an obvious second chip with limited code, processing and pins, when your can add a co-processor to the silicon of the BMC (either legit made in China or a clone slipped in to supply chain) if you're being cheap, you would add a second die during packaging and tap on to the bonding wires.

    This is critical news for every "friendly" country and all business, but we can't be arsed to show any proof, even dodgy dossier WMD level chicanery.

    All part of the info wars going on, just hope it stays virtual.

    Tossers, the lot of them, nearly all of them in nearly every country, those who think they're better than the plebs, fcuking people over for their own ends. A pox on you all.

    1. Uffish

      Re: Second chip added, bollocks

      "all part of the info wars going on" Upvoted for that - whether you are right or wrong.

  38. MacroRodent

    Article: "Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one?"

    Who knows, maybe this has also been done on some other motherboards...

    1. Ommerson

      Presumably because the contents of the SPI flash are easily verified- and something more sophisticated customers would actually do.

  39. Anonymous Coward
    Anonymous Coward

    Iiiii don't know… SEC and FBI should investigative Bloomberg (the company and the person) for market manipulation (I wonder how much they shorted on this news) and destabilizing Western society (there must be a Russian link somewhere here). Of course, heads should roll and sanctions imposed.

  40. Mystic Megabyte
    Unhappy

    Trump is trying to divert attention from his Russian master, look over there---> It's Chinese hackers!

  41. Anonymous Coward
    Anonymous Coward

    MSG

    Is this going to turn out like the fake MSG scare stories...?

    "That Chinese stuff: bad!"

  42. Norman Nescio Silver badge

    Taiwanese foundries

    I imagine some people may decide it is beneficial to source hardware from Taiwanese foundries, given that politically, that area is not mainland Chinese, Russian, or American/5-9-14-eyes controlled. Of course, you would still need to ensure that your supply chain has not been subverted in some way, given the example of the NSA and Cisco.

    Assuring that your hardware has not been Trojanned is a hard problem, which the vast majority of people don't need to worry about. I have every sympathy for those whose job such assurance is.

  43. Doctor Syntax Silver badge

    So many birds with one off the record briefing stone. Draw attention from whatever exploits you're making with Intel ME. Prepare the way for "Nice little motherboard business you've got there. Pity if something happened to due to manufacturing off-shore/not co-operating with us. Remember the Super Micro incident".

  44. naive

    Does it matter if the Bloomberg report is true or not ?

    Maybe it was indeed all orchestrated fake news, the scenario itself is however very probable in case we not aware of potential supply chain issues.

    Parties who have to rely on foreign IT components, be it hard or software, should take into consideration to whom they want to be exposed, there are only two or at best three choices, except for Europeans, who do not make much IT gear, so everyone is free to spy on them.

  45. The Dot

    Amazon's response

    https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/

    I generally find Bloomberg to be a decent source of biz/finance news. But for IT stuff, they know 2 things, jack and shit.

  46. Wildrex

    How do we know this isn't just us Government making up crap because sanctions are not going to work

  47. Anonymous Coward
    Anonymous Coward

    Perhaps the most surprising thing about all this is learning that there are BMC chips that have their firmware stored on a physically separate chip, with exposed connections between the two, and nobody thought that this might possibly represent a vulnerability.

  48. Someone Else Silver badge

    "layers of lawyers"

    Kieran, I love the alliteration, but...EWWWWWW!

  49. tmz

    What am I missing here?

    "A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors."

    How is the contractor/attacker in any way in control of which MBs get sent to which customers. Surely that is entirely in the hands of Super Micro? Are these particular compromised MBs only available to certain customers? If so, how?

    1. Malcolm Weir Silver badge

      Re: What am I missing here?

      I think you're missing the section in the Bloomberg piece about how Supermicro is well known for customization...

      My analysis is that these attacks were targeted at SKUs specifically known to be of interest, not the general SKUs that you see on the Supermicro website.

  50. Calleva65

    One questions seems to have been missed

    Why was Elemental targeted?

    I haven't even seen anyone question why they would be targeted or what they do.

    In fact AWS Elemental are a Cloud and on-premise Video processing, storage and monetisation company - i.e. they are in the TV world.

    For the life of me I can't see why China would be that fussed about getting access to what TV show is being played on a random four letter TV station.

    Does this mean that the BOFH-on-the-street should be more worried that a fundamentally random company got these motherboards?

    Or, I am sure some will say, that it was to get into Amazon - so the Chinese knew that Amazon might want to buy Elemental before they did in order to design and insert components, with all required connectivity, not just the chips, sell them and have them shipped and used before Amazon started their due diligence. That would be impressive!

    1. doublelayer Silver badge

      Re: One questions seems to have been missed

      As I remember, elemental had a contract with an American intelligence agency, although I don't think they're still doing that. However, that could have made them a target at the time.

    2. Wzrd1 Silver badge

      Re: One questions seems to have been missed

      For the life of me I can't see why China would be that fussed about getting access to what TV show is being played on a random four letter TV station.

      Well, FEDRAMP is hosted largely on AWS, so that isn't exactly a random four letter TV station, it's the primary access point for civilian US federal agencies cloud presence.

      As for bloke on the street targeting, I know a number of peers, as well as myself who could be targeted, due to the PRC hack of OPM and the downloading of our security clearance investigation files.

      One upside of that is, now, we can get a security clearance in China.

      From the course views of chip locations and traces, it's probable these are CMOS wedge devices, pre-pre-execution environment for the BMC, root kitting it at a hardware level, before the BMC CMOS gets loaded. That bypasses checksums, signing, etc.

  51. Milton

    You'll never buy bullets from China

    I agree it's impossible to be certain who is accurate/misled/mistaken/lying through their teeth. Unless you have actual knowledge of this event, first-hand, you are guessing.

    But here's the thing: this is an obvious and highly effective means of espionage, for which a highly technicallly capable nation state, one which lacks checks and balances, and which is well motivated to spy upon foreign governments, militaries and corporations, and which is an ever-increasing source of computers and computing components to the rest of the world, is the perfect source. China, in short, has both powerful motive and ample means.

    Consider that no one with serious security concerns brings a computing device back from China (or if they do, it's quarantined, stripped, analysed and then incinerated). Consider that China's spyware has been busy for many years siphoning data from western firms and governments. Consider that even consumer grade devices have been found 'phoning home' with personal data from their owners. If any nation could build the necessary hardware into a speck 100-μ on a side, no thicker than a hair, would you seriously bet against China's best? And bear in mind that China is desperate to become the world's next hyperpower, and that arguably only US technology and economic strength could stop them.

    Add it all up, and whether the SuperMicro story is total bollox, or 100% true—it hardly matters: of bloody course the Chinese will be trying this kind of trick, and it would be frankly amazing if they haven't already succeeded here and there—and perhaps already on a large scale. How many devices get national security examination or Amazon/Apple level audit? How many ways, how many places, how many disguises could there be for a cleverly designed sequestration/exfiltration nanobug?

    Off the cuff, I'd say that for the next few years this battle is already lost. Amazon will probably say anything to deny that its cloud has been compromised, but I remain confident in saying that if you trust vital data to anyone's cloud, you are a fool.

    I said a while ago that in due course, nation states and their allies will bring in-house the manufacture of hardware and software for critical components and infrstructure. Expensive as it is, what choice can there be? Soon enough, computing components will be like ordnance. You may make missiles yourself, or you may buy them from the USA; but you'll never buy them from Russia. Or China.

  52. David Pearce

    China along with several other nation states are quite capable of dissecting the Intel ME and discovering how to remote control it, no need to modify any hardware

    1. ROC

      All they need is to brush up on Minix....

    2. Stoneshop
      Holmes

      no need to modify any hardware

      You know, belts AND suspenders.

  53. Anonymous Coward
    Anonymous Coward

    So they've found the first spy chip.

    How many more to go?

  54. Anonymous Coward
    Anonymous Coward

    What about Quality Control

    Having worked in the industry on thing pops out at me. Supposedly security agencies caught this. All production has Quality Control. Visual inspection is done, at least on a spot check basis. One checks to see if there are any parts that have fallen of during manufacturing, shock and vibe. A missing capacitor, no matter how small would be caught. Given that this happens, any extra chip would also be flagged by Quality Control. I could see quality control catching this and forwarding the issue to security agencies, not the other way around.

    To add this chip it would have to be to intercept the board in shipment ad replace it, not in manufacturing. This seems like a very unlikely scenario to me and therefore I lean towards Super Micro, Apple, and Amazon statements being more credible.

    1. tmz

      Re: What about Quality Control

      From the Bloomberg report:

      "A U.S. official says the government’s probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack."

      QC at Supermicro (or its documentation) would be the number one target for this, I would have thought.

  55. Marco van Beek

    Right at the end of the Businessweek article it says "In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge"

    Even based on the little we do know, that is bollocks. Elsewhere in the article they say "the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected". So there is a commercially viable way of detecting the chips. Good old-fashioned traffic monitoring.

    Sounds more like all those old chain emails that used to go around about viruses that "nobody could detect", encouraging you to forward the email on to as many people as you could to warn them. GCHQ and NSA probably have enough taps on enough lines to do this for us.

  56. JaitcH
    WTF?

    The Chinese Have Come a Long, Long Way Since Chairman Mao Encouraged the Establishment ...

    of small backyard steel furnaces in every commune and in each urban neighbourhood.

    Perhaps Trump should quit bitching about the Chinese stealing US technology and do a bit more stealing of his own, this time Chinese technology.

  57. Graham Cobb Silver badge

    Are SuperMicro systems going cheap?

    I have been thinking about buying a server and sticking it in a CoLo, for offsite backups. Maybe now is the time?

    I don't care about the SuperMicro share price -- are their servers going cheap now? I don't care if the Chinese can copy my data -- the rest of the world has it already thanks to my government.

  58. Anonymous Coward
    Anonymous Coward

    I'm not worried, all they will find on my system are pictures of naked vegetable, phoar look at those onions without their skins on.

    No nuke sub schematics here.

  59. Cynic_999

    Occam's Razor

    If China had the means to design and manufacture such complex "spy chips" then it would also have been able to manufacture lookalike replacements for legitimate chips on the board such as the BMC chip itself. This would not only have been more difficult to detect (as there are no unexpected additions or changes to the board), but far easier to implement involving fewer people. All that would need to happen is to substitute the bogus chip for the real chip as supplied to the manufacturing factory which could happen anywhere along the supply route, or been done via a "black bag" operation substituting the stock in the warehouse of either factory or supplier. Nobody at any factory need be aware of any changes. No highly difficult modification to the PCB layup (putting a chip between fibreglass layers would require a different and completely non-standard manufacturing process for the PCB - everyone working at the PCB factory would know what's going on).

    It simply makes no sense that such a highly complex and detectable method involving scores of people would have been used when a simple component substitution would have done the job far better and cheaper with far less probability of detection and no 3rd parties ever needing to know that it has happened.

    1. Ommerson

      Re: Occam's Razor

      If you're in the business of verifying that the motherboard contains what it should you'd hi the parts off, mill of the packaging and check the die with a verified sample. Perhaps the genius of a component masquerading as a passive, is that nobody would bother.

      1. Cynic_999

        Re: Occam's Razor

        "

        mill of the packaging and check the die with a verified sample.

        "

        If the spooks are capable of making their own silicon as the allegation suggests, then the die itself can be made to look little different to the genuine product. In many cases a complex chip contains its own CPU and microcode - the only difference between the real and bogus chips being the microcode in it's on-chip ROM which will show no visible difference.

    2. Anonymous Coward
      Anonymous Coward

      Re: Occam's Razor

      "... been able to manufacture lookalike replacements ..."

      Maybe they have. This device is just the decoy.

    3. Stoneshop

      Re: Occam's Razor

      a highly complex and detectable method

      Depends on who you want it undetectable (or at least nearly undetectable) for. On the manufacturing side you actually need just a few people who know the details: the ones modifying the schematic and the PCB layout, and creating the manufacturing manifests for the board etching/sandwiching/populating machines. It's the ones that handle the boards after they've shipped (building them into systems, reflashing, further inspection, etc.) that these mods need to be hidden from.

      Also, for everybody else in the manufacturing chain these are just like any of the other customised boards destined for a particular customer.

      1. Cynic_999

        Re: Occam's Razor

        " ... and creating the manufacturing manifests for the board etching/sandwiching/populating machines ...

        "

        This is like saying that a car manufacturer could start making flying cars without any of the factory workers noticing except those operating the machine that bolts the wings on. PCB manufacturers do not *have* any machines for sandwiching chips between layers. It is not a normal PCB process. I think all the factory workers would know about a brand-new machine and completely different workflow.

  60. Jaybus

    No Worries

    You would have to be an idiot to have Super Micro's BMC accessible from the Internet.

    1. Stoneshop
      FAIL

      Re: No Worries

      And you would have to be an idiot to believe that it won't work with the BMC not having a direct internet connection.

  61. wsanders

    The US has fabricated intelligence information before to start a war. I think there is just as high a probability as anything else that this is fabricated, or exaggerated, and Bloomberg has been duped, just like most major US media outlets and Congress were duped into going all-in for the Iraq war.

  62. Anonymous Coward
    Anonymous Coward

    Frankly I'm amazed the cloud guys even bother with BMC

    Surely AWS don't bother with a baseboard controller on their servers...i.e. go to the aggravation of allocating an IP, a subnet etc just so some lucky NOC noggin can web or SSH in once in 18 months when the server seems squirrely? I mean their volumes make this deeply unlikely - you'd need hundreds of people to monitor the server estate in that manner. I posit that AWS's ask SuperMicro for custom server designs that don't even bother with the chip.

    So while I can well believe the Chinese might attempt something like this, I'm also skeptical that the attack exists as described.

    And of course, are we really to believe that the Chinese assumed that AWS's networks would simply allow the rogue chip to phone home for instructions? Those things must be locked up tighter than a gnat's sphincter.

    To reiterate: yes I believe the Chinese state has motive, means and opportunity. And as they say on the UK cop shows I watch, they "have prior". I'm just not convinced about this particular attack.

    1. Malcolm Weir Silver badge

      Re: Frankly I'm amazed the cloud guys even bother with BMC

      Err... plugging a cable into the IPMI port and then let it grab an address with DHCP is not exactly the most complex provisioning task. By contrast, the NOC noggin is far more likely to incur higher costs by accidentally prodding the wrong box than the total cost of the cables, switches and infrastructure required for this.

  63. Anonymous Coward
    Anonymous Coward

    Sad to say, but this story is probably true...

    1) Everyone knows that China loves to acquire/steal/copy other countries intellectual property. They aren't the only ones who do this, but I think I am safe in assuming that most Regenistas would agree that they at least have the public reputation of being the worst actor in this area. There is a reason why if I were to use the phrase "Chinese knock-off" when talking about a product, pretty much every Regenista would pretty much know what I was saying

    2) Like every sigint/intel community, the Chinese want to know what target governments/militaries/companies/organizations are doing.

    3) Unlike most sigint/intel community members, the Chinese have a very large portion, and perhaps most of the sub-assembly and component supply chain in their own country.

    4) The Chinese government is very authoritarian, and will openly throw up regulatory/legal/political roadblocks against companies that don't do what the government wants. I've seen this first-hand, and of course there are barge-loads of news stories about this.

    5) The Chinese culture also has a higher-than-normal tolerance for bribery.

    6) Thanks (once again!) to Edward Snowden, we know that both the U.S. and British intelligence communities have the capability, operational authorization and much past experience with intercepting and backdooring electronics shipments when they are in the supply chain. It is pretty much certain that the other members of the 5 Eyes agreement either have this capability in-house, or they subcontract this kind of work through relevant requests to their U.S. and British partners.

    7) Given their reputation for corporate and governmental espionage, I would assume that at the very least France and Russia also have and use these capabilities in-house.

    So why wouldn't huge, powerful and increasingly global China do this? It's not like China is morally like a giant version of Sweden or Switzerland. They have no compunctions about being very tough and aggressive versus potential regime opponents, ethnic and religious minorities, journalists and whistle-blowers, companies that want to invest in or export to China, neighboring countries that are not overt allies, etc.

    So given that China and others are almost certainly doing this, then I guess the real issue is what do IT and tech professionals do about it? One, make sure that you have elaborate network monitoring, data and firewalls if you are dealing with strategically valuable or sensitive commercial or government information. Your average small/mid-sized business can get away with basic antivirus and email filtering, but once you are dealing with data where you can look at it and objectively say "I could see how country X would like to get this info to give an advantage to their government or corporate sectors.", then you should probably start pressing your management to get very serious about network security.

    Two, give as little cover as possible to national security/intel agencies that try to get involved in standards-setting. Assume that they do not want secure IT for anyone but themselves. In fact, truly secure IT would tend to dramatically reduce their usefulness to political and strategic decision-makers, and therefore negatively impact the future of those agencies and the careers of their employees.

    Three, if you are dealing with the kind of valuable information that I discussed above, and you have the resources to really audit and inspect your hardware, do that.

  64. Bitsminer Silver badge
    Thumb Up

    A couple of other points

    ...because similar motherboards were in use "in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships."

    Uhhhh, nope. Chinese-manufactured motherboards cannot be sold to US governement agencies, especially military or intelligence. Stuff has to come from one of a trusted list of countries called TAA (Trade Agreements Act, FAR 52.225-5). China is not on this list.

    Just try to find a disk drive not made in China. Thailand maybe? Been there, had to find that. Did.

    ..."the middlemen would organize delivery of the chips to the factories."

    Wow. Just wow. This is classic misdirection. The Intelligence folks are trying (and largely succeeding with Bloomberg, el Reg, WaPo, etc) to focus attention on one vulnerability, namely surreptitious factory modifications. But there are more, so many more. A few hints:

    o Connectors. Yes, those boring black thingies with wires going in, and out. Embedding a chip within a connector requires no BMC changes, is difficult to check even with Xray, and completely unobservable. And since there is exactly one connector model that fits in exactly one place, no wastage. Profit!

    o Firmware "adjustments" as many others have suggested. But where? Not just SMI flash but ... power supply flash. RAM controller firmware. CPU firmware. Or simply radio transcievers embedded in the board (or connector) that introduce firmware changes at boot time. (Where is the transmitter? Hmmm...maybe power supply RF emissions?)

    o Known zero-day vulnerability in....CPU (obviously), BMC firmware (also obviously), but: Ethernet chips, memory controller chips (like RowHammer), PCI bridge chips, etc

    o NSA black bag ops. However, at the scale of Amazon, Apple, etc, probably not cost effective. Except see my next point.

    One thing not mentioned by El Reg, is the scale of procurement by Amazon, Apple, etc. Unless merely for development purposes, these companies purchase servers by the container-load (where the container is pre-loaded with racking, switches, power, servers, etc). Thousands of servers per container. The assembler might be persuaded to mung things up. It seems a remote possibility but the supply chain risks at this point (well after the motherboard factory) have not been addressed in the press that I can see.

    +1 for El Reg and very well reported. Thanks.

    1. Steve Chalmers

      Re: A couple of other points

      Wonder if there's a peer exploit putting a little cell phone in, say, the power supply, to get to the outside world from an air gapped system. Yeah, I know, a bit harder in a shielded room, but sure seems like thinking one could "phone home" with IP traffic is a little optimistic if the target had any sophistication or was of any value at all.

  65. goldcd

    What a lot of FUD

    If Super Micro construction had been compromised, well you'd think one of these boards would have escaped to an outraged owner who'd have provided a photo of the 'unknown chip'

    As it stands "There's been a massive hack" - and then "a massive coverup" - and all involved wish it to go away (except for a load of third parties who've both leaked the details and hidden the evidence)

    1. razorfishsl

      Re: What a lot of FUD

      Clearly you have little understanding of the problem.

      If you ever get the chance to take the top of a "Real" computer......... good luck on identifying even half of the components without highly specific engineering drawings.

  66. shaolin cookie

    When the two sides stories sound credible...

    ... they are likely to be misled by a third party. So who stands to gain from making such noise? About how the US shouldn't be buying Chinese stuff but make their own, and if things happened to go wrong, the media is dishonest?

    Especially given the timing just before the midterms, would be highly surprising if some at the GOP were not behind this story. I mean if the story was real, what are the odds of the loudmouth administration actually keeping quiet about something that would drive their agenda perfectly?

  67. Malcolm Weir Silver badge

    I reckon this is a cautionary tale

    As others have noted, where are the modified boards? Why are seeing pictures showing some random small thing next to a pencils/pennies? How do we reconcile the denials against the story?

    As has been pointed out, an attack like this is plausible. It could be done. It might have worked if it was done. But it alternatively might have been detected early, and that detection resulted in nothing happening.

    So my working hypothesis is that this is a cautionary tale: beware of your supply chain.

    And for that, it doesn't matter (to the teller of the tale) whether all the details are 100% factual, because they're just there to jazz up the story. Apple, Amazon, a bank, CIA spy videos... even Supermicro. The point of the story is not that Something Happened (to Apple/Amazon/the bank/whoever), but that Something Could Have Happened.

    Consider this: is it plausible that *if* China was surreptitiously tinkering with a motherboard that they would have succeeded first time out? Why does the Bloomberg article talk about various different types of spy device, without ever explaining why there are the variations?

    So we have a report of several generations of spy chip with no explanation of what the second and subsequent ones were for (they can't still be for the Amazon boards) and how they were identified, and we have no exhibits of the compromised items even though there are several generations of spies implying several generations of targets.

    And we have NO information about the "phone home" mechanism which is, apparently, teh whole point of the spy chip. And I'm not just talking about the absence from the Bloomberg article: NO ONE has publicly reported and described suspicious activity resulting from the nefariousness. No security notices have been released suggesting a list of IP addresses to block. And, err, there are non-public infosec channels that exist to disseminate advice to people in the US defense industrial base, and if there was corroboration, I'd expect it to leak.

    So I think this is a hypothetical attack being reported as an actual attack at the behest of the US government as part of an effort to help prevent actual attacks using this sort of technology.

  68. CrispyD

    Logistically implausible

    From the Bloomberg article:-

    """

    “Having a well-done, nation-state-level hardware implant surface would be like witnessing a unicorn jumping over a rainbow,” says Joe Grand, a hardware hacker and the founder of Grand Idea Studio Inc. “Hardware is just so far off the radar, it’s almost treated like black magic.”

    """

    The real question in all this is: Who is the logistics super-genius running all of this, and when are they going to get poached by Amazon?

  69. boidsonly

    Amazon and Apple are not going to chew off the hands that feed them. I place more faith in the Bloonberg report than their denials....

  70. razorfishsl

    You only have to look at poor FDTI and their range of RS232 chips.

    Tell me none of those went into servers and other equipment.

  71. DanielR

    The very people making the accusations like the NSA have been caught planting malware into Cisco hardware. The BMC have their own port and embedded operating system and web console seperate from the motherboard. I had one myself for a Sun server they called it Lights Out Management.

    Nobody in their right mind would have this unprotected without behind a vpn / firewall. I can tell you this gets brute forced.

    The level of effort for little gain is a dead give away. If the BMC isn't even connected then this hack is useless too. I am not sure if it's part of the motherboard or a daughterboard as the Sun server's was.

    I can only presume it's seperate and "isolated" although it gives you full access to the server terminal like a keyboard and monitor. To configure the bios and power cycle etc.

    But what is for sure they think because it's a remote console, and mention malware plant, they think people will automatically believe them. I want to see packet inspection logs !!

  72. wizodd

    Truly irrelevant whether or not the story is true--it is a warning that we must heed, that our electronic civilization is subject to manipulation by others.

    We tend to delay security until we are burned, partially due to cost considerations, and partially plain failure to understand that a risk still exists even if no cases have yet occurred.

    Billions of dollars worth of software has been written offshore by programmers not restricted by law from stealing it and perfectly positioned to make unknown modifications. Such code changes are seldom easily spotted if at all. Similarly forged chips can contain code and unknown functions.

    It is long past time we abandoned passwords and 4-digit PINs for anything of any real importance, and time to recognize and build protections into our sourcing systems against spyware hard and soft.

    If we await our first "big" loss, the cost might destroy us.

    1. Charles 9

      "It is long past time we abandoned passwords and 4-digit PINs for anything of any real importance, and time to recognize and build protections into our sourcing systems against spyware hard and soft."

      Ever heard of the phrase "the cure is worse than the disease"? At least the "big" loss isn't a guarantee, but killing ourselves trying to guard ourselves from a risk may well be a certain death. In which case, it's better to just roll the dice. After all, we've been fapping around with passwords for decades...because there just isn't any better alternative that employs nothing but the human brain, especially for faulty brains. AFAIK, it's a physical impossibility: what man can create, man can re-create. It's a problem that's been known since before World War II (based on the writings of E. E. "Doc" Smith, who had to resort to science fiction to find a foolproof solution).

  73. Anonymous Coward
    Anonymous Coward

    Did anyone actually verify any motherboard?

    Why there's tons of speculation on who's right and who lies instead of actually getting samples of those MBs and actually inspecting them. I understand Bloomberg might not have experts, but there are plenty of security companies in US and Europe.

    They should be itching to get their hands on samples and tell the whole world they actually found it and know its insides.

    Believe no one. Check the facts.

    1. Charles 9

      Re: Did anyone actually verify any motherboard?

      From what I've read, they're supposed to be custom boards, meaning they're under contract and probably considered trade secrets. IOW, samples won't be forthcoming due to legal trouble.

  74. wownwow

    Is SEC still hibernating?

    It caused the SMCI stock to drop more than 40%! Is SEC still hibernating?

  75. Anonymous Coward
    Anonymous Coward

    There's a weirdly large amount of downvotes on the comments, including this one.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like