back to article Everyone screams patch ASAP – but it takes most organizations a month to update their networks

The computer industry may have moved to more frequent software security updates – but the rest of the world still takes a month or longer to patch their networks. That is one of the findings in a new report by enterprise network bods at Kollective. The biz spoke to 260 IT heads in the UK and US about their systems and security …

Page:

  1. Anonymous Coward
    Anonymous Coward

    "Windows 10 will automatically update your system"

    WSUS and proper group policies achieved it since XP/2003 at least. I have groups that are automatically updated when critical patches are issued - it something bad happens, we'll rollback, we assessed the risks and costs of unpatched system was higher than the cost of rolling back if needed.

    Others may need manual approval, depending on what happens if a patch has issues.

    The biggest issue under Windows is the software which requires to be updated manually because it requires a specific procedure often only partially automated.

    Other systems has patch issues - i.e. Python virtual environments, if the base Python they are created from is patched, the venv may not...

  2. Pascal Monett Silver badge

    These posts are very interesting

    Most of them point to patching issues with large or very large user bases.

    I am self-employed. If a patch borks my system, I am good for a full day of reinstalling everything to be able to work again. Who's going to pay me that time ? Nobody.

    If my system is bricked, then I am good for an emergency trip to the nearest quality hardware dealer and a hefty ticket price to get a new machine, which I then have to spend the day cleaning, removing stupid vendor-installed cruft I couldn't care less about, and getting the stuff I need to start working again. So a day lost again, and a big expense that I have not budgeted. Who's going to pay me for that ? Nobody.

    If my system should be hacked (has never happened), then at worst, I'm good for losing a day reinstalling.

    So my threat profile tells me that I can wait a while before patching, to see if there are any howls of pain from the latest batch of Windows updates. If I don't hear anything for a few weeks, then I put Windows Update back into Auto and patch, reboot, finish the patches and reboot again. Then WU goes back to Disabled, where it belongs.

    1. Anonymous Coward
      Anonymous Coward

      "Then WU goes back to Disabled, where it belongs."

      You can set it to simply warn you when patches are available, maybe download them, but not install until you tell it to (of course, if you're not using Windows 10).

      No need to disable it fully - some critical patches may be released outside Patch Tuesday. Not all patches are also system patches - patching IE or Office is far less risky than patching Windows kernel - still, they can ensure protection against dodgy contents (IE could still be used by applications using it embedded, even if you don't use it).

      For example, I easily postponed for a while the Meltdown patches because there were issues with some Asus motherboard management tools.

      Patch that totally bork a system luckily are not that frequent, and may bork only specific systems, and still there could be ways to rollback without a total reinstall. While a ransomware hitting you could be far worse - a borked OS disk is usually still readable....

  3. Drew Scriver
    FAIL

    Fear and pride...

    Management is commonly driven by (mainly) two factors: fear and pride.

    Apply that combination to any project or service and the chance of success are greatly diminished.

    Pride drives hasty releases ("Watch me meet deadlines!"), a preference for the latest-and-greatest ("I'm hip and modern"), results in jumping on the latest bandwagon ("Always ahead of my golfing buddies"), cutting costs ("See me stay under budget!") - you name it.

    Fear drives hasty releases ("If I miss the deadline I'll be in trouble"), avoids patching ("I'm not going to be the one who causes stuff to break"), doesn't enforce standards and requirements ("The best conflict is the one avoided"), and so forth.

    Many companies therefore have created an environment where patching is all but impossible. Rather than saying that compliant applications are a requirement, all app-owners, vendors, and the like have to test and sign off - and each has the power to halt patches, even if only one application out of hundreds might break if, lets say, SSL3 is disabled...

    Of course, if they don't have the time (or knowledge...) to test their application they won't be able to sign off on the required patch, and fear then drives the decision to forego patching.

  4. mutin

    Patching? Actually it should b Vulnerability Management cycle

    Auto patching system may fail. I've seen that. In particular funny (sorry) was that I explained IT director that vulnerability scanning is standard way to check if patching works. He was not idiot but ... So, when they hired IT boss on the top of him and we resumed scanning, we found that 30% of computers were not patched while system reported they were. Then virus outbreak happened on the top of multiple vulnerabilities.

    So, patching we discuss here IS NOT THE GOAL. It should be always Vulnerability Scanning after.

    Is it possible to do within a month? Very hard. Almost impossible considering complex IT systems. The only one success story was when I did VM for Navy installation of 4,000 computers ten years back. Somehow IT guys managed to patch and I was able to do my scanning. Since that I see only sad stories.

    The chaos result of of what we have now was created by IT giants rushing for profit no matter what.

    They created the environment of "IT jungle" where we - the food for predators and them aka "hackers" - will coexist forever. The only one way to limit your risks is to limit your Internet connections. Pack your bag, forget your computer and go South. Bingo.

  5. Paul Hovnanian Silver badge

    Is it a patch?

    Or are you trying to shove Windows 10 onto my machine?

  6. Chris Evans

    Please enlighten me

    When patches are issued they normally give quite detailed information about the problem they are fixing. Why are so much details given at this stage?

    Many patches I read of seem to have not yet been exploited so why give the hackers many of the details they need?

    I would have thought those details would not be released for say a month, though reading this article makes me think three months would be better!

    If the cat is out of the bag and the issue is being exploited already then I understand there is no point in delaying things.

    1. It's just me

      Re: Please enlighten me

      Because even though you may not be able to apply the patch immediately, if you understand the details of the vulnerability you may be able to determine that your configuration doesn't have that vulnerability or expose it to attackers or that there is a mitigation that you can quickly put into place that will protect you until you can apply the full patch.

      Even if they don't give details, that doesn't stop the hackers from finding out what was fixed, for example, using binary diffs and disassembly.

      MS has stopped giving details on their patches and just pushes out a few big ones that may contain dozens of fixes. That hasn't stopped the criminals from releasing exploits soon after.

  7. JeffyPoooh
    Pint

    Blatantly obvious solution...

    When Microsoft (or whoever) issues a new patch, the IT staff should instantly roll it out to their selected several victims. Then, on the odd occasion when the freshly-patched systems malfunction, the IT staff can wander over, blame and ridicule the hapless users for their misuse of the systems and surfing illegal dodgy websites causing such crashes, all while quietly uninstalling the recent patches and rolling back the system. After a few days of this not happening, they can start rolling out the patches to more and more users - all within about a week at most.

    This should be the de facto approach.

  8. MAH

    Microsoft has made patching a complete cluster with these stupid monthly rollup patches.

    Seems every month they break something, but you really can't exclude the one broken component out of 20 so you have to skip the patch. the next month, they fix that broken component, but now a different component is broken so now you wonder which broken component is worse.

    Look at July rollup....41 serious issues with the rollup...including the .net which broke sharepoint, exchange,etc. If Microsoft can't even develop a patch that doesn't screw up their own inhouse applications (which should be really simple for them to test right) who trusts them not to screw up every other vendors applications.

    It comes down to this, Microsoft has completely lost everyone's trust when it comes to patching because they don't bother to test at all (which is obvious with the july .net patch) so no one wants to just set auto updates and go....

  9. Anonymous Coward
    Anonymous Coward

    In theory, and practise

    Currently, I'm a sysadmin of the firm I work for, and a number of our customers.

    Regardless of the network set up, bandwidth, automation, best practise and all the money/time in the world thrown at it, MS still struggle to release TESTED patches.

    Scenario : Patch Tuesday comes around, MS release 60 patches for Office, 20 for Servers 2008 -> 2016 (applicable for x64 infra), and Adobe throw in some for good measure. These get approved when they appear on WSUS.

    End Result : Patches get rolled out, finance complains because scripts to integrate the financial system and Excel for reporting doesnt work anymore, web services stop working because you're running them compliant to .NET 3.6 but they rolled out .NET 4.2 and your other software companies haven't yet updated their application to work with it.

    Sysadmin spends a couple of hours patching, several hours unpatching, meanwhile getting blamed for being out of scope/GDPR compliance on system security and integrity.

    OR

    Everything goes through fine, no problems reported, but all of your clients stop reporting to WSUS because the patch wasnt tested with WSUS deployment (NO, Not everyone wants to use InTune/SCCM!!) and now you have to manually patch all the clients to get them reporting again.

    Oh and dont forget, auto deployment of the 6 month updates to Windows is bad for on call users. Best to do that manually.

    As above, sysadmins seem to get a lot of the blame and responsibility when we are only responsible for the maintenance and upkeep - not actually developing these patches.

    And don't get me started on testing procedures in small companies....

  10. Roland6 Silver badge

    Kollective get top marks for the misuse of survey results

    There is a distinct lack of clarity about just what is being talked about. I see no real connection between "a critical remote-execution bug in Apache Struts 2" ie. 'datacentre' systems, and the way an organisation may go about fixing this and relying "on employees updating their own systems" ie. end user systems.

  11. Okole_Akamai

    I've done patch management for 15 years now for organizations with offices around the globe. Here's what I've learned:

    1. Don't break anything.

    2. It is a collaborative effort to distribute patches in an Enterprise environment. More importantly you need Executive support to do it right.

    3. Test the patches, document results, compare results, establish a schedule, obtain buy-in to move forward. If something isn't right or breaks, speak up.

    4. Communicate results, ensure you have buy-in to distribute patches and risk is accepted by delegated authority.

    5. Don't be the person everyone glares at when you come into the office the morning after a patch distribution.

  12. Lorribot

    What causes delays? Systems designed to finish a project and not to be managed. websites that fall over when their database server reboots. Having servers in the hundreds or thousands so only option is patch automatically, but applications need to be shut down gracefully before patching and that can't be scripted. Applications that need to be logged on as a specific account and run a specific application on start up (yes really in 2018 they still exist) SQL server that have multiple databases on and patching the apps in the right order and sorting it all out is beyond 5 minutes work. There is much more.

  13. SAdams

    To do fast patching reliably, you really need to have (the equivalent of) Winrunner and Loadrunner scripts setup on your pre-peoduction environment that are constantly maintained for all critical applications, and then a full team to manage all these scripts and update them each time an application, OS or middleware is updated. However now that most companies use VM’s on replicated storage, as long as the storage has snapshots (and there is some failover mechanism), security should really take presidence when roll back is an option.

    I suspect most companies with *nix patch less than monthly ?

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like