back to article Azure VMs borked following Meltdown patch, er, meltdown

Microsoft Azure customers are reporting problems with their virtual machines, which are struggling to come back online after being updated with the Meltdown processor patch. The firm told one customer in a direct message on Twitter: "We are currently investigating alerts affecting Virtual Machines in West Europe. You should be …

  1. Anonymous Coward
    Anonymous Coward

    I wonder why, as I read that Windows was successfully patched in November.

    1. fnusnu

      Only for those on the insider track

    2. Anonymous Coward
      Anonymous Coward

      Looking at the Azure support Twitter account and Azure support forums it doesn't seem to be that widespread an issue.

      1. Anonymous Coward
        Anonymous Coward

        Azure was down but it doesnt say it was

        Notice the Azure Twitter has a bunch of questions and Azure is moving those conversations to private? I have contacts in North America that were down as well as Europe. Yet azure status site and history shows nothing. https://azure.microsoft.com/en-us/status/history/

  2. Karlis 1

    > This is down for 8 hours and still no ETA. Is this the kind of support we should expect from Microsoft.

    yes?

    1. Robert Moore
      Holmes

      > This is down for 8 hours and still no ETA. Is this the kind of support we should expect from Microsoft.

      yes?

      If you are not expecting this level of support from Microsoft, you have not been paying attention.

  3. DCFusor Silver badge
    Trollface

    Heh

    Obviously, Rust would have prevented all this. /sarc

  4. PhilBack
    Mushroom

    VMs not coming back have not been waiting for Spectre or Meltdown to happen

    Got biten for weeks with these problems on Azure West.

    Not going there anytime soon.

  5. Steve Davies 3 Silver badge
    Mushroom

    Yay! Welcome to the Cloud

    Welcome to the Future.

    You are part of the borg collective.

    You and all your data is now owned by the likes of Microsoft.

    Oh, and 'test' is a 4-letter word and is like the 'dead' and the 'dodo'.

    1. Doctor Syntax Silver badge

      Re: Yay! Welcome to the Cloud

      "Oh, and 'test' is a 4-letter word"

      User testing proceeding according to schedule.

  6. Dwarf Silver badge

    Should we be worried

    Aren't MS preparing a similar patch for various machines enslaved with Windows OS's *

    * Except probably versions we care about and that they don't

  7. J J Carter Silver badge
    Trollface

    BOFS! What are they like?

    BOFS managed to cock-up their cloud-hosted VMs! A bad workman always blames his tools.

    1. hplasm Silver badge
      Facepalm

      Re: BOFS! What are they like?

      "BOFS! What are they like?"

      Literate?

  8. Anonymous Coward
    Anonymous Coward

    I bet Ballmer is laughing into his cornflakes this morning reading this on his iPhone

  9. rmason Silver badge

    like mnany I suspect.

    Like many others i'm up patching tonight.

    mixed CentOS and windows environment, haven't had anything fail to come back up *yet*.

    Worth noting for others though;

    Our benchmarking stuff is showing a 20%+ (ish)slowdown on postgres (SQL) on centOS.

    I've had tests at everything from -7% on some boxes, to the -20% mark. Bad times.

    1. Anonymous Coward
      Anonymous Coward

      Re: like mnany I suspect.

      Just a thought, if it doesn't run code from the outside and if you don't want the performance hit - then why patch?

      I've got something performance critical, and I probably wont be patching them (haven't tested the difference, yet).

      1. Pax

        Re: like mnany I suspect.

        This is part of what organisations should be doing in assessing the risk to their organisation on machines which don't run user interactive sessions, and "adequate protection" is deemed to be in place.

        Can your business application/users cope with the performance hit, and can you scale your resources accordingly to mitigate a performance hit?

        If you are in public/hybrid cloud, what is the cost in horizontally scaling resources to mitigate performance hits.

        1. Maventi

          Re: like mnany I suspect.

          > This is part of what organisations should be doing in assessing the risk to their organisation on machines which don't run user interactive sessions, and "adequate protection" is deemed to be in place.

          Good call.

          > If you are in public/hybrid cloud...

          Then it's probably best to take the performance hit as you never know who else might be sharing your compute node with potential access to your own host's memory.

      2. Adam 1 Silver badge

        Re: like mnany I suspect.

        > if it doesn't run code from the outside and if you don't want the performance hit - then why patch?

        In the case of Azure, you are not going to be running on bare metal. You are going to be on a VM guest, so the important question is whether your kernel's data can be read from collocated VMs belonging to other customers. I am personally unclear on whether patching the VM host is sufficient or whether both host and any guest need both be patched.

        But yes, if you have a machine which is air gapped with a performance critical workload then you are one of the lucky few.

        I am sure we won't be waiting too long for this to be exploited via JavaScript or a PDF/docx/xlsx file with some macro. That is going to suck big time.

    2. Alistair Silver badge

      Re: like mnany I suspect.

      We're seeing almost 60% of a single core being eaten by our networking -- this is on a hadoop data node. 14 cores, but still likely to cause some imbalance long run.

      We're trying some tweaking to see if we can make it better. (closed system testing in a sandbox, no way I'm putting this in prod yet)

      1. rmason Silver badge

        Re: like mnany I suspect.

        To try and answer some questions:

        I didn't patch everything that night, this was a "suck it and see" test on playground servers running copies of both our internal stuff, and the stuff we sell. We will be patching everything though, regardless of performance (they must then be "fixed" somehow).

        Like most places we are hybrid, some AWS and azure, some on prem stuff, it's all getting patched, that is a clear directive from "on high".

        I Imagine the future fix for performance will involve upgrades both to physical and cloud stuff, and improvements by our devs to the product.

        I'm not touching prod stuff in terms of the product, because of the variance in issues seen on our SQL driven stuff, but the windows stuff that's on prem (DCs and FS basically) is all patched up (as far the update produced so far) and done.

  10. Captain DaFt

    Another week, another "Azure Borked" headline.

    They do seem to come around with amazing regularity, don't they?

    Did Microsoft hire the Swedish Chef to oversee Azure?

    1. Naselus

      Re: Another week, another "Azure Borked" headline.

      No, but they may have hired the guy who was doing processor instruction ordering at Intel in '95 to do so.

  11. Tezfair
    Unhappy

    might not be just VMs...

    Customer server installed something last night and since then hourly throws this error...

    ----------------------------------------------------------------------------------------------

    File Server Resource Manager Service error: Unexpected error.

    Error-specific details:

    Error: GetVolumeInformation, 0x80310017, The data drive specified is not set to automatically unlock on the current computer and cannot be unlocked automatically.

    ----------------------------------------------------------------------------------------------

    It only started appearing in the logs after an update. They do have a bitlocked backup, but that has opened ok and the other drives are fine.

    1. TheVogon Silver badge

      Re: might not be just VMs...

      That's nothing to do with this update.

      Removable data drives must have either a password or a smart card unlock method in addition to the automatic unlock method. Automatic unlocking cannot be directly specified by policy settings.

      To configure a BitLocker-protected fixed or removable data drive to automatically unlock, follow these steps:

      1. Click Start, click Computer, and then right-click the BitLocker-protected fixed or removable data drive that you want to automatically unlock.

      2. Click Manage BitLocker, click Automatically unlock this drive on this computer.

      1. Tezfair
        Thumb Up

        Re: might not be just VMs...

        bitlocked backup drive are open and fine. Not been an issue for a few years then started after last night. But appreciate the imput.

  12. Anonymous Coward
    Anonymous Coward

    No problem for our VM:s in West Europe...

  13. elip

    Nah, we're in multiple regions, and they're all experiencing slow-downs in IO. Not isolated to W Europe. I also love how the patching/Maintenance status updates on our VMs are taking between 4-6 hours to actually show the current status. Quality stuff.

  14. Anonymous Coward
    Anonymous Coward

    Solution switch to AMD not Meltdown patch

    Its a bit obvious....Intel is broken

  15. Anonymous Coward
    Anonymous Coward

    There are always these surprised articles when MSFT has an unpatched vulnerability or issues a patch and everything goes haywire... have the people writing these just come across Microsoft. Par for the course.

  16. Anonymous Coward
    Anonymous Coward

    Remembering Snowden....

    Yup.....Meltdown and Spectre are both causing trouble.

    But how long before we hear about ALL THE OTHER ZERO DAY PROBLEMS which the NSA, GCHQ, the Russians, the Chinese, and all the other bad actors have squirreled away against the day that Meltdown and Spectre end up patched everywhere?

  17. Ace Solo McCloud

    I'm waiting for the inevitable NHS article... they'll either get hit because they didn't patch, or their systems will grind to a halt because they did patch.

  18. portyman

    cloud and hyperconverged

    In the rush to all things new and "flashy" in the assumption its cheaper a lot of people have been burned.

    many years a go someone said, don't put any data into the cloud you don't want other to see. We had cloud providers saying how secure it was but strangely went very quiet when we asked if they would guarantee that with some money should data leak out, Never heard a salesman stumble of words so quickly.

    Public cloud will never be secure, every security feature that has been created, has been got around at some point, or will be. Things are too complicated these days making it almost impossible to make things secure.

    Hyper Converged was touted as the solution to most data centre issues, Spectre has shown that sharing your data and compute on the san node is suicide for security. I hated the idea anyway due to the state of software testing these days never mind trying to get different vendors to fix issue.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019