back to article Patch blues-day: Microsoft yanks code after some PCs are rendered super secure (and unbootable) following update

A bunch of PCs running the wares of Sophos or Avast have been freezing or failing to start following the installation of patches emitted by Microsoft on 9 April. The afflicted are those running Windows 8.1, 7, Server 2008 R2 and Server 2012. Avast for Business and Cloudcare have been hit by the problem, as have PCs running …

  1. Anonymous Coward
    Anonymous Coward

    So? It's merely anti-malware code doing its job..

  2. David 132 Silver badge
    Coffee/keyboard

    Wonderful phrasing

    Thank you for the wonderful phrase ”firehose of whiffy code sprayed from deep within the bowels of Redmond.”. Made my morning.

  3. Christopher Rogers

    Is it a Microsoft failure or a Sophos failure?

    1. Halfmad Silver badge

      More to the point why is only Avast and Sophos borking? I can understand MS AV products running along happily but the other security products would surely have thrown a fit too when Sophos did.

      Makes me think that some aren't as secure in some areas, particularly boot as they claim.

      1. RAMChYLD

        I would think it's the way Avast works that is at fault. It certainly didn't like it when I installed Visual Studio 2016 on my machine and started locking the system up at boot. Getting rid of it for Avira solved my issue.

        1. richardcox13
          Headmaster

          > Visual Studio 2016

          I would expect that to trigger any MalWare as fake software.

          Of course if you meant 2019...

      2. big_D Silver badge

        We have Kaspersky's and Symantec's corporate offerings and no problems so far (couple of hundred PCs and Servers patched before the warning came out). I rolled it out to a test group, then a day later to all PCs, after there weren't any problems.

    2. Anonymous Coward
      Anonymous Coward

      Sounds like Sophos failure. They are tapping into something that they not suppose to be tapping into.

  4. Alister Silver badge

    Well thanks, I could have done with this article yesterday, instead of chasing round the office rolling back updates.

    1. Gene Cash Silver badge
      Pint

      New patch test procedure: "wait a couple days to see if an El Reg article comes out"

      1. Claptrap314 Silver badge

        That's been SOP for decades, son.

      2. DMSlicer

        Honestly? That's pretty much spot on these days for corporate PCs...

        1. Patch Tuesday happens, making a bunch of new updates available on the MS Website.

        2. Our WSUS server syncs with the new list of updates sometime early Wednesday Morning (4am ish)

        3. Client PCs sync with WSUS sometime on Wednesday, and tell it what they need from the new list of updates. WSUS then goes off and actually downloads all the needed updates in the background over the course of the day from the MS Website onto one of its local disk drives.

        4. Client PCs sync to WSUS again every ~6 hours. At some point whenever WSUS has their requested updates available, the clients will start to download them. Group Policy determines whether the local user of the machine is actually permitted to install them or not (which kicks in at midday - so realistically this will be 12pm Thursday or Friday depending on the number of updates needing downloaded), otherwise they wait for them to be manually installed by IT staff during an agreed downtime window.

        If at any time there's any argy-bargy noticed on the interwebs relating to botched updates, those updates get declined on WSUS before Thursday midday; so they never actually hit the client PCs.

        It's literally a built-in time delayed "stop those updates!" killswitch.

        This week we actually got told about the AV Crashing Issue on Thursday at ~11am, and I had a Decline in place on WSUS by 11:15. And a lucky escape it was too, since we're one of the poor fools whose higher-ups have mandated the use of Sophos Enterprise...

  5. Zog_but_not_the_first Silver badge
    IT Angle

    Can't argue...

    A unbootable PC is "super secure".

  6. muhfugen

    How retarded is the author, that he thinks that this is somehow Microsoft's fault for these programs which hook undocumented API calls?

    1. Tabor

      I agree that my first thought was “another AV vendor that fucked up” as well, but at the moment of posting it’s not clear what caused the issue.

      Calling the author “retarded” is a bit much though, even if it’s only related to 3rd party software.

    2. el kabong Silver badge

      "...hook undocumented API calls" huh? Links please...

      I assume you learned that from a reputable source, could you please share the details.

      I hope they're not hidden behind a paywall.

    3. Terry 6 Silver badge

      EXCEPT. Microsoft will know these products well. They aren't obscure. So they should have been tested for.

      Over the years their has always been the excuse of "well we can't possibly test every possible configuration of software/hardware...."

      And it's always been at least partially bullshit. Because they screw up on standard configurations that most users are likely to have.

      1. Anonymous Coward
        Anonymous Coward

        I learnt pre Windows NT but you can't trust a hardware compatibility list.

    4. bombastic bob Silver badge
      Meh

      "these programs which hook undocumented API calls"

      back in the 90's, MS got sued over having undocumented API calls that Word and other MS applications leveraged in order to "not crash all of the time". I had just started doing windows development for Windows 3.0 and ran into some of these problems with 'GlobalLock' etc. which could occasionally UAE when you had a bad handle, not necessarily your fault either.

      The solution, for me, was to validate the param by using the undocumented 'GlobalHandleNoRIP()' and fail gracefully. At the next dev conference, Microsoft introduced the 'ToolHelp' library [to assist with things they'd been doing internally, cough cough] and DOCUMENT all of those undocumented calls...

      [they also separated the applications from the OS as separate 'business units' and complied with 'no information between OS and applications units that is not disclosed to EVERYONE']

      So - with respect to 'undocumented calls' - which ones are those again? Microsoft may be forced to DOCUMENT them...

  7. x 7

    causing havoc in parts of the NHS, thankfully not in mine

  8. AMBxx Silver badge

    t's best to try them on a test machine before allowing them anywhere near production

    Useless advice - are we now supposed to keep parallel, identical computers running? What about non-standard software that's been installed?

    I don't know if we should blame MS or the AV vendors, but either way you can't have umpteen test PCs.

    1. el kabong Silver badge

      Blame both.

      I'd wager they are both at fault, I like to stay on the safe side.

    2. Rich 11 Silver badge

      Re: t's best to try them on a test machine before allowing them anywhere near production

      Useless advice - are we now supposed to keep parallel, identical computers running?

      I do. They're not identical in terms of content or scale but they are identical in installed software and general configuration. They're close enough to be a canary in the coal mine, for my software development QA as well as for vendor patches and upgrades. I accept it might not be achievable in everyone's environment, and certainly not even in all of mine all of the time, but I always aim to get such an arrangement agreed in the original project plans and budgeted for. If the project manager or the beancounters turn it down and something subsequently goes wrong, my arse is covered.

      1. Bronek Kozicki Silver badge
        Linux

        Re: t's best to try them on a test machine before allowing them anywhere near production

        I do that too. They are pleasure to work with because they boot really quickly so testing is simple, let me build the updates myself (if I wish to) and the license is not only free, they also let me see and tinker with the sources or build options.

        Oh ... but that gives the game away, doesn't it? It's all Linux. Smug, me?

        1. Rich 11 Silver badge

          Re: t's best to try them on a test machine before allowing them anywhere near production

          Yeah, smug you. ;-)

          It's fine if you've got the option, but if the business requires you to work with Windows servers (or whatever) then you've got to work within those requirements and solve problems within (as well as with) those requirements.

          1. Hans 1 Silver badge
            Windows

            Re: t's best to try them on a test machine before allowing them anywhere near production

            It's fine if you've got the option, but if an idiot requires you to work with Windows servers (or whatever) then you've got to work within those requirements and solve problems within (as well as with) those requirements.

            TFTFY

    3. big_D Silver badge

      Re: t's best to try them on a test machine before allowing them anywhere near production

      We have a bunch of "test" PCs, they get updates and patches ahead of the hoi polloi. They are single PCs in key departments. If the updates don't cause problems, they are rolled out to the rest of the machines.

  9. Anonymous Coward
    Anonymous Coward

    Testing ?

    What happened to the send to the test users, to pilot users and then to prod a few days later to check there are no known problems / issues ?

    Pretty standard when I used to do global patching

    1. Alumoi

      Re: Testing ?

      Didn't you get the memo? Now the paying customers are all testers for Microsoft.

  10. Lee D Silver badge

    Stop installing patches on the day of release automatically.

    Set up a small set of machines, in a separate WSUS group, apply them there, test them there, then roll out to larger and larger groups as you get more confidence in them.

    Seriously, "auto-update as soon as something is released" is the stupidest thing I ever heard.

    Home users, sure, I understand that they don't have the nous, but even so a "I know what I'm doing" button should always be present.

    You cannot forcibly inflict updates on any group of people, though, unless they've CONFIRMED that they have a backup ready and in place. That MS think they can just force it anyway, without even telling the user, and just assume they've been backing up is RIDICULOUS. Hell, even System Restore is pretty much a waste of time that I don't know of anyone ever successfully using to any decent extent.

    1. Alister Silver badge

      In this case, System Restore has been the saving grace for us, as the only way back for 20 plus machines.

      A symptom of the problem which I've not seen mentioned elsewhere was that on a full boot, after CTRL-ALT-DEL no login box showed up, so users could not get into their machines at all.

      Booting in safe mode, and doing a system restore to a restore point prior to the upgrade has been how we've fixed the issue.

    2. elvisimprsntr

      My employer uses third party tools to manage and push out updates after they have been vetted, rather than clients pulling updates directly from M$.

      1. Lee D Silver badge

        If you have Windows server, you have WSUS. No need for third-party (though third-party often does it more nicely).

        There's no excuse, one you have more than a handful of computers, for not using WSUS for exactly what it's intended for - scheduling updates, revoking them, and centralising all Windows Update downloads in your organisation.

        1. Anonymous Coward
          Anonymous Coward

          Although you are correct I think that only distracts from the problems caused by MS updates. MS's new(ish) rolling update system does not seem to fit for purpose, but I'm sure if saves MS more than enough money to continue contributing to Senators and Congresspeople as they find expedient.

          I'll bet those contributions don't come cheap.

  11. Herby

    Microsoft's QA is working...

    ...just as planned. Welcome to beta test.

    1. big_D Silver badge
      Black Helicopters

      Re: Microsoft's QA is working...

      It is a subtle hint, didn't you notice? Only the OSes on life-support were affected.

  12. revenant Silver badge

    Interesting set of victims

    My understanding from the article is that the systems affected were running Business or Enterprise AV software on earlier versions of WIndows.

    So this problem specifically hits businesses that haven't moved up (?) to Windows 10.

    Hmmm.

    1. OssianScotland Silver badge

      Re: Interesting set of victims

      Server 2008R2 and 2012 were mentioned, so I don't think a Win10 upgrade will be a solution

    2. whitepines Silver badge
      Devil

      Re: Interesting set of victims

      Surely you mean "downgraded to Slurp Edition?"

      Hey, I like it! Let's call it Windows 10 SE !

  13. Numen
    Meh

    W10 + AMP = '!'

    On one of our Windows 10 systems, running Cisco AMP, after the Tuesday patches went on we get a yellow '!' on the Windows Defender shield icon. When you look, it says "Virus and threat protection status unavailable, open Cisco AMP for Endpoints for information." The link it provides is to the AMP Connector, which won't open. Opening AMP directly, it thinks things are fine. I'd turn on AMP Connector Debugging if I could find where the log file for this is!

    Anyway, something seems off in the Connector interface.

  14. Pascal Monett Silver badge

    It's just the next step

    Before, we had all those Windows Update patches to avoid that every knowledgeable person had a list of. That worked not very well, so Microsoft is on to the next round of the same, but with a different excuse.

    The goal remains the same : get them all up to Slurp Edition.

  15. sequester

    Certificate revocation?

    If Microsoft shipped an update revoking code signing certificates for the software causing the problem, wouldn't that solve the problem? System-level crapware services would no longer start, and everything would be fine.

    1. whitepines Silver badge

      Re: Certificate revocation?

      And who do you think would be sued when the machine gets a virus? Sure, the EULA might protect Microsoft...but then again, an average judge and jury might understand "Microsoft silently / remotely turned off the antivirus and the victim got a virus.". It's not rocket science. Even worse that victory might start undermining the other aspects of the EULAs....probably safer just to brick the machine, everyone understands updates break things, right?

  16. Anonymous Coward
    Linux

    And the solution being ..

    And the solution being to install Linux.

  17. Anonymous Coward
    Anonymous Coward

    ill make sure to watch out in a month

    people update to prod in patch tuesday week? Jesus no 1 month behind MS always. SCCM is a bulky beast but serves a purpose.

    And just Avast and Sophos affected, hmm doesnt sound like a real windows issue to me, symantc/Webroot rocking here and in global corp space, sounds more like Soph&A not following MS best practise for such tools, yet its Ms fault. ;o)

  18. Geekhillbilly

    Another Attempt to force a Win 10 downgrade

    I run neither Sophos or Avast but my Windows 7 64 bit setup was rendered unbootable by the April 9 update, forcing me to restore using the Monday system image backup. I disabled Windows update, figuring that this update was another attempt to force me to use the Windows 10 Spyware OS. As of now, I am preparing to move to Linux to get away from Micro$oft's BS.

  19. rmstock

    Happy Death Day 2U

    Are we sure it's not straight from the movie script ?

    "an unfortunate user has downloaded "the thing"

    [ .. ]

    And of course Windows 10, to which Microsoft would dearly like users to upgrade, is not affected by the borkening.

    Thanks to all the Reg readers who got in touch about this.

    We feel your pain. ® "

  20. Biggered

    Windows 10 & Avast

    Windows 10 has also been affected by this (1803 & 1809), during testing I've had issues on 3 W10 devices, which magically worked normally again once the update had been uninstalled in Safe mode.

    This was caught on the W7 devices on the first test device and reported to Avast (via Barracuda who now own Managed Workplace) and so hasn't gone any further.

    Avast have pushed out a micro update which needs to be installed and have a restart before the update then stops causing problems.

    Sophos look to be working around with exclusions.

    Fun all round....

    I shall no doubt be moaned at for taking so long on updates this month!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019