back to article Enjoy the holiday weekend, America? Well-rested? Good. Supermicro server boards can be remotely hijacked

Tens of thousands of servers around the world are believed to be hosting a vulnerability that would allow an attacker to remotely commandeer them. The team at Eclypsium says it has discovered a set of flaws it refers to as USBAnywhere that, when exploited, would potentially allow an attacker to take over the baseboard …

  1. Duncan Macdonald Silver badge
    FAIL

    How many ?

    How many of the administrators of the vulnerable servers even know that the system has a Supermicro motherboard ?

    How many of those even read security vulnerability notices ?

    And how many of those know how to update firmware ?

    And of those how many will bother ?

    My guess is that at least 50% of the currently vulnerable systems will still be vulnerable in a years time !!!

    Icon for the administrators who do not do the security updates ============>

    1. fnusnu

      Re: How many ?

      How many of the vulnerable servers even have a system administrator?

      1. ds6 Bronze badge
        Mushroom

        Re: How many ?

        Ours switched to USM and refuses to do T2 work in the interim :-]

    2. phuzz Silver badge
      Facepalm

      Re: How many ?

      Whoever hooks the management port up to the internet, without even putting a firewall in the way, kinda deserves whatever happens to them.

      Mind you, whoever does that probably has equally terrible security practises everywhere else, so a BMC issue is the least of their problems.

      1. oiseau Silver badge
        FAIL

        Re: How many ?

        ... without even putting a firewall in the way, kinda deserves whatever happens to them.

        Indeed ...

        But if the server has Intel's Management Engine (ME) or AMD's similar AMD Secure (the usual scenario), I think there's no firewall available to protect you.

        O.

        1. LeahroyNake Silver badge

          Re: How many ?

          There may be no firewall if you are on the same network segment but I highly doubt there isn't at least a rudimentary firewall on the 'Internet connection'. Then again some plonker could have just forwarded all ports to the system with the vulnerable BMC... I doubt anyone would be that stupid though :0

        2. This post has been deleted by its author

          1. FIA

            Re: How many ?

            I want to like AMD BTW (even though I've been fucked by them hard with the HP laptop of fire, going mad when that broke and the fan wasn't awlays on on the next laptop, not even needing a book to have it on ALL THE TIME because idling on a surface will cause it to shutdown for thermal reasons, the fx-8350 "8 core" crap ect)

            You're mad at AMD because HP can't design laptops??

            FWIW the intel based HP laptop I once had was a piece of shit too.

            Really interested in the new Zen stuff, just trying to get over aforementioned fucking

            Apart from being released too early (the early motherboards were a bit.... fun) the Zen stuff is really good. Plus being able to swap my 1600X for a 3xxxx and get the performance boost is a boon too.

            Been building computers for years, had similar issues with both Intel and AMD over the years. Have had some complete rubbish, but also some amazing stuff, in both camps.

            It wasn't the fastest, but my bought on day of release 'Bulldozer' box is still going strong; My current i5 box is a bit 'funny' in some situations, whereas the previous core 2 was bullet proof. The first Zen box I got was a bit flaky, but after a drunken accident took it out I bought better quality stuff and that's been super stable ever since.

            1. ds6 Bronze badge
              Coat

              Re: How many ?

              What's the OC on the 3XXXX like? I bet it gets hot enough to burn a hole in the carpet.

              Mine's the one with thermal paste on the sleeves.

        3. Ian Michael Gumby Silver badge
          FAIL

          @oiseau Re: How many ?

          I'm sorry but if you put a firewall in place and the ports are not open to allow traffic... no problem.

          Or am I missing something?

          1. Michael Wojcik Silver badge

            Re: @oiseau How many ?

            Or am I missing something?

            Well, you're missing the fact that most successful IT attacks against organizations of any significant size involve penetrating the corporate network through any one of the zillions of holes already punched in it - dodgy WiFi routers, employees with VPN connections and careless browsing habits, SSRF attacks, and so on - and then pivoting and escalating.

            USBAnywhere is a pivot-and-escalate dream come true.

            The "egg model" (corporate network hard on the outside but soft on the inside) has failed. While it protects from a huge number of everyday attacks, it nearly always fails quickly when confronted with a dedicated, knowledgeable attacker. Even security firms (e.g. Hacking Team) get taken by penetrate / pivot / escalate chains, often by lone, amateur attackers.

      2. Solviva

        Re: How many ?

        Often by default the management port piggybacks onto the regular Ethernet port, it's up to you to restrict it to the dedicated ME porf (if it even has one)

    3. Solviva

      Re: How many ?

      Our Supermicro supplier up to magbe a year ago used to ship boxes with the default login and dhcp enabled... So the minute it found power and an ethernet connection it was wide open - you didn't even need to hit the power button.

      Machines destined for me got fixed pronto (so I could install an OS from my desk),but I realised for other machines, their 'owners' perhaps didn't even know they had IPMI!

    4. atpage

      Re: How many ?

      Well then patch it for them.

  2. Captain Scarlet Silver badge

    Query

    Do Supermicro boards have a seperate port for the ip kvm functions?

    Most servers I have used have dedicated ports for the ip kvm but I do know of some HP servers sharing NIC1 with the IP KVM.

    1. Dedobot

      Re: Query

      Have few X11 - all of the with dedicated 100mbs lan.

      The ports mode can be switched but on default they are dedicated bmc management.

    2. Alan Brown Silver badge

      Re: Query

      Depends on the model but in general yes - however by default ipmi will jump to the other ports if the dedicated port isn't active.

      ALWAYS _check_ your ipmi settings before letting a server loose on the network.

      Of course some helpful individual could connect to all those open ones and issue a "power off" command.

      1. Captain Scarlet Silver badge

        Re: Query

        Ah so basically a trying to be useful feature which might come back and bite anyone not aware of it.

  3. Dedobot

    I cant see any SM's fault.

    IPMI device just exposed online is beyond stupidity.

    1. big_D Silver badge

      A security flaw is a security flaw. Whether the admin adds to the problem through lousy configuration just exacerbates the problem.

    2. LDS Silver badge

      Even if it's accessible only inside a network something into the network can be compromised in other ways, and this becomes another easy prey for lateral movements and persistence, even more so because it's on systems mostly used as servers, and some probably as network appliances as well.

    3. rcxb Bronze badge

      How about the worm potential? One internet-facing server may have a root compromise, then the attacker can use the attached BMC to jump onto the fully segregated BMC/IPMI network, and start attacking ALL the OTHER servers to continue its spread, and gain access to higher-value servers?

    4. Michael Wojcik Silver badge

      If you think all attacks come from outside the corporate network, you're not doing much better.

  4. Andy Taylor

    One other thing to be aware of is that by default, if the IPMI network interface is not physically connected at startup, the BMC will share the connected interface.

    This can be disabled using the ipmicfg tool.

  5. Khaptain Silver badge

    Port forwarding necessary

    I can't see how this can be done without port forwarding between the router and the server and or some similar form of NAT/PAT.

    Or are there really people that mount servers 'Directly' connected to the Internet... Really, it's actually not so simple to do.. You would have to have more than just basic knowledge in order to do so, and if that was the case you would presumably know about the risks involved... Or am I missing something here?

    1. Paul Crawford Silver badge

      Re: Port forwarding necessary

      IPv6? Enough addresses so everything can have its own global internet address!

    2. LDS Silver badge

      Re: Port forwarding necessary

      I wouldn't be surprised if some bright mind did exactly that to allow for "remote manglement", especially some small installation with no on-site admin, and no VPN to access the internal network... or "just in case if the VPN doesn't work".

      It does impact some "embedded" micro-systems (i.e. the SYS-E300 series) as well, that may have been configured that way easily.

    3. Tabor

      Re: Port forwarding NOT necessary

      I can see how. Compromise one target in a network, pivot from there. Plenty of networks out there that don’t separate vlans or that don’t use firewalling between those vlans.

      It IS disturbing though that so many seem to have the interface exposed to the Internet directly.

      1. Michael Wojcik Silver badge

        Re: Port forwarding NOT necessary

        It IS disturbing though that so many seem to have the interface exposed to the Internet directly.

        I don't know that's any more disturbing than the widespread misapprehension that internal networks are magically safe.

    4. SImon Hobson Silver badge

      Re: Port forwarding necessary

      Or are there really people that mount servers 'Directly' connected to the Internet...

      Yes, it's not that uncommon. NAT is an abomination to be avoided if possible. At a previous job, we had a whole /24 to ourselves (legacy from when addresses weren't scarce) and all servers were on public IPs (IPv4). We did have a firewall between them and the internet though ;-) Go to any "pile em high and sell em cheap" web hosting provider and you'll find your server is on a public IP.

      Really, it's actually not so simple to do

      Actually it is trivial to do IF you have the public IPs. It's actually easier than putting them behind NAT.

      Even if you only have the one IP and need to use NAT, it's usually easier to forward "everything" to one internal device than it is to setup lots of port forwarding rules. Port forwarding is more flexible, and offers less scope to mess up the firewall rules - such as failing to change the default "allow everything" rule in a Draytek.

    5. streaky

      Re: Port forwarding necessary

      Really, it's actually not so simple to do

      It's absolutely trivial if you give it a public IP.

      Also by the way there's billions of reasons to use IPMI not just related to no onsite admin - if you run thousands of servers you want a way to manage them, to provision them - it's the public IP *plus* flaws that's at issue here. They're both fixable.

  6. iron Silver badge

    And this is why the Bloomberg rice-sized spy chips on mobos story was a lie. Why go to all that trouble when you can just find a flaw in the manufacturer installed management engine.

    1. Yet Another Anonymous coward Silver badge

      Unless of course that was just a trick by the Chinese MMB to distract you from this vulnerability. Which is really just there to cover up the real spyware

      1. Anonymous Coward
        Anonymous Coward

        Nope.

        It was the lizard people trying to distract you from the crab people who were trying to distract you from the alien invasion taking over Madagascar.

        1. Yet Another Anonymous coward Silver badge

          Re: Nope.

          No I'm in the illuminati, we get a newsletter about lizard people

          1. Michael Wojcik Silver badge

            Re: Nope.

            Some newsletter. Half of it is just glossy ads for human skinsuits. I quit reading it years ago.

  7. Jay Lenovo Silver badge
    Trollface

    You see! Oh.. it wasn't that

    Darn it, I thought we'd finally found the work of those Chinese spy chips.

    1. DougS Silver badge
      Trollface

      Yep, though it turns out

      The spy chips were made by Intel, not China, and they have a name: "BMC".

  8. Anonymous Coward
    Anonymous Coward

    I do love a good USB hack!

    From looking at the packet capture linked to in GitHub it looks like it's using the standard emulated CD iso like used in the old U3 enabled flashdrives back in the Windows XP days.

    isolinux.bin

    Also, I see several references to Samsung in the capture?

    0x0050: 2020 2020 2020 2020 5361 6d73 756e 6720 ........Samsung.

    1. the spectacularly refined chap

      Re: I do love a good USB hack!

      Well, yes, that is what it is designed for.

      While there are clearly some vulnerabilities here I can't help but feel that it is being over-egged in significance. In substance it amounts to an authentication bug, screaming "look what I can do" is hardly evidence of further flaws when those are documented features of the system.

      Bear in mind the wider context here: these are professional grade server boards. People then pay a premium for these BMC equipped boards. If you opt to use a shared rather than dedicated IPMI port one of the first things you are asked is what VLAN you would like the remote managent on to keep the traffic segregated. Even if you never use it (possibly because you bought an inappropriate board in the first place) the vulnerability doesn't arise because it needs a prior login during that power cycle and before the BMC is rebooted.

      So yes, there is a flaw here which has been patched. But do not start down the "dumb lusers don't understand this" road: it doesn't affect them and if for some reason they are using what are advanced professional-grade tools it is ultimately their responsibility to mitigate the well understood risks of these fetures.

  9. Kobblestown

    MIA

    SM does a good job at acknowledging the flaw but the actual firmware for my Xeon E3-12xx line, X10-variety mobo is still missing. Anyone found updated BMC firmware for such boards?

  10. T. F. M. Reader Silver badge

    Let's sum it up

    "BMCs ... allow admins to ... perform critical maintenance tasks, like updating the OS or firmware" +

    "BMCs ... aren't typically designed with security in mind" +

    "Sep 3, 2019" = ?

    No, does not compute.

    1. Anonymous Coward
      Anonymous Coward

      Re: Let's sum it up

      Let me help:

      BMCs are designed to be connected to physically isolated internal networks, or at the very least a separate dedicated non-routeable VLAN on an internal network.

      It was mentioned in the article, in passing, that BMCs are rarely connected to the Internet. That is why. Yes, it would be much better if this crappy firmware were fired into the sun and replaced by a quality open source product designed from the ground up with a sensible security model in mind. Yes, it would be much better if operators demanded that Intel and AMD provide the necessary documentation to properly write and install such software into the BMC and other system components in place of the delivered binary-only firmware. Yes, it would be much better if this software could be customised to disable unneeded components to reduce the attack surface. No, that's not what's been happening, and everyone knows this stuff is total garbage which is why it's never connected to a globally routeable network. It's not an excuse, just a fact. You can use a steak knife to cut off your hand, but they're designed to be used to cut food and everyone knows that.

      1. LDS Silver badge
        Devil

        'BMCs are designed to be connected to physically isolated internal networks'

        The code Supermicro uses even supports dynamic DNS....

  11. Nate Amsden

    SM IPMI still terrible

    Shouldn't be surprised I suppose.. my last SM IPMI update(~5 years ago) part of the instructions was to wipe the configuration, I suspected that included the network configuration meaning it would no longer be accessible on the network after rebooting. But I tried anyway just in case. And sure enough yes the IPMI went offline at that point and I didn't have connectivity to it again for another couple of years (next time I went on site, fortunately had no HW failures in the meantime). Add to that the terrible documentation SM has on when firmware is updated, what is fixed etc(release notes seem more common for them on their newest stuff from the looks of it).

    I replaced my personal SM server (which otherwise worked OK as in no failures anyway, my SM experience goes back to about 2001) last year with a Dell R230. For work stuff historically I use (since ~2006 anyway) HP, but in this case HP didn't offer a configuration that I wanted so went with Dell. Has worked well so far anyway. My personal server is at a co-lo and runs a half dozen VMs, though maybe will add more VMs got tons of capacity now.

    Back to SM..

    Since this article mentioned "X10" I wanted to see what the current situation is, so I poked around for an X10 board with IPMI

    first web hit was this board:

    https://www.supermicro.com/en/products/motherboard/X10SLM-F

    seems recent "Single socket H3 (LGA 1150) supports Intel® Xeon® E3-1200 v3/v4, 4th gen. "

    downloaded the IPMI firmware package, and at least in this case they give release notes and a list of fixes, but in the "IPMI Firmware Update_NEW.doc" file they still say in big red letters(had higher hopes given the "NEW" in the name)

    "NOTE !!! Uncheck preserve configuration box during flashing (very important step for FW to work properly). All settings will be reset to default."

    I suppose if you are using Windows, DOS or Linux on the bare metal that may be ok, but for me anyways running vSphere there are (as far as I could tell anyway) no vSphere related tools for IPMI config on supermicro.

    At a bare minimum there should be an option where you can at least populate some basic configuration such as network configuration so you can connect to the IPMI after it resets. Hard to believe this situation is unchanged years later. I have had seamless upgrades on HP iLO and Dell iDRAC (used Dell at a company back in 2009-2010 too) on every single attempt, not a single issue over the past ~13 years. Before that I was mostly a SM customer (had a few hundred systems at one point) and firmware updates were basically never applied as the process and documentation was quite scary(I believe often required DOS floppy disk on systems that had no floppy drives, and the remote KVM/virtual media abilities did not exist at the time), and SM themselves warn you not to upgrade anyway(still warn you even today). Their processes and documentation are only marginally better today.

    A while back I looked at the IPMI update procedure for Citrix Netscaler, and it was just horrifying https://support.citrix.com/article/CTX137970 ), I believe Citrix uses Supermicro as well. I tried once getting IPMI working on Citrix but ran into a wall pretty quick(I think the certs were the same on every device which caused browsers to freak out, known issue at the time anyway), just use serial console and network PDU.

    1. rcxb Bronze badge

      Re: SM IPMI still terrible

      the IPMI went offline at that point and I didn't have connectivity to it again for another couple of years (next time I went on site

      Wait... WHAT?

      From within the OS of a running system you can access the attached BMC/IPMI to change the network settings, reboot it, etc. Whatever you need to do.

      1. LeahroyNake Silver badge

        Re: SM IPMI still terrible

        The poster did suggest that they were running a hypervisor on the system. I would hope that it would prevent any client OS running on it from accessing the base hardware?

        They also mentioned the lack of tools for Vsphere so they have looked into it.

        1. rcxb Bronze badge

          Re: SM IPMI still terrible

          running a hypervisor on the system

          Fair enough, I didn't make it quite that far through the wall of text.

          I would hope that it would prevent any client OS running on it from accessing the base hardware? They also mentioned the lack of tools for Vsphere so they have looked into it.

          VMWare/ESXi has a "management console" that is basically a Linux VM but with privledged access. On the management console, you can run ipmi configuration utilities. Here's a fellow who shows how to do so with SuperMicro specifically:

          https://www.cryptomonkeys.com/2016/12/supermicro-ipmi-reset/

  12. Anonymous Coward
    Anonymous Coward

    This Story Over At Bloomberg...

    Big story. Massive. They went all out reporting this one.

    Errrrr.

  13. Boy Quiet

    It seems elemental that any organisation would have at least one outer DMZ where all ports are closed by default (block all) and specific rules to open required ports.

    If the organisation takes credit card payments in the UK, it’s a PCI requirement and no QSA should sign off a site without checking the actual firewall config not just the policy document.

  14. Kobblestown

    Jesus Fucking Christ guys! Has anyone been able to locate fixed BMC firmware for any board? I couldn't do it for mine (it lists an older version) but then I browsed around and couldn't find it for any board! WTF is going on?!

  15. E 2

    IDK

    IDK about this being a huge issue qua security hole.

    If a server's BMC is accessible from the public Internet then the server's owner has worse problems than what are described in this article.

    This seems like some of the "holes" described last year where the attacker has to achieve root level access to a box to exploit. If an attacker has root on your box then how likely are they to exploit obscure holes rather than just do bad stuff, stop syslog, clean up the logs and restart syslog?

  16. Ubermik

    Isnt this a bit like saying "if you leave the door to your safe open people who are allowed into your home and are THEN left alone in the room with your OPEN safe "might" take something from it, therefore you shouldn't buy a safe"....

    What makes this even more silly, is how many people have pointed out that server admins might not even realise which servers have these boards in them without hitting network inventory data. Yet "someone" from, I guess CHINA "might" somehow magically arrive in a companies server cluster room and would then somehow magically know what the companies own techies dont and would somehow get physical access to the server to exploit the issue.....

    TBH, this just seems like a less moral facet of the US trade war with china, you know, like the BS over the Huawei products JUST after they "coincidentally I am sure...." overtook APPLE in global phone sales with an even better model JUST about to launch.....

    What tech companies SHOULD have taken from the Huawei fiasco is that NO company in ANY country is safe as long as it relies on US hardware, software or firmware

    ARM needs to move completely out of the US so it can sell its chips where ever the heck it likes and other tech companies around the world need to work on having open source, openly traded equipment with no link to the US

    Then the rest of the world can carry on advancing whilst the US clings desperately to its ever decreasing stranglehold on the worlds tech industry

    Leave US tech companies ONLY selling in the US market and to its owners in Israel with everyone else selling EVERYWHERE else

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020