back to article Dodgy Kaspersky update borks THOUSANDS of NHS computers

A misfiring antivirus update from Kaspersky Lab "effectively levelled several networks" last months including systems at Britain's NHS, The Register has learned. A Kaspersky update released on 25 October falsely classified a Windows system file, specifically tcpip.sys, as malign and quarantined it. The procedure left Windows …

COMMENTS

This topic is closed for new posts.
  1. TonyJ

    What happened...

    ....to testing updates before they get applied?

    It's no major issue if an update borks my own laptop - I can rebuild it and restore as necessary but I'd expect controls to be in place before any system-wide updates went out for an enterprise.

    1. Richard 26

      Re: What happened...

      "....to testing updates before they get applied?"

      You would think any IT operation of reasonable size would stage definitions a bit, I agree. What I would like to know is why enterprise antivirus deployment tools are happy to roll out buggy deployments to hundreds of machines, borking them one by one without blinking an eye?

      Is it so hard so roll out updates to a small number of machines, see if they still work afterwards, then make an intelligent decision on what to do next? Even doctors are smart enough to work like that.

      1. DaLo

        Re: What happened...

        How many companies, let alone NHS trusts do you know that have the budget and resources to employ an AV update bork-checker?

        Updates come out for AV at least once a day and sometimes more often. To load this into a test network and try out every bit of software to see if it still works okay for that once in 5 year time that it doesn't is unrealistic.

        In an ideal world, yes it would be great but you expect your AV vendor to have the resources to do testing for at least major software (like the WIndows OS) to make sure it is not affected.

        1. codeusirae
          Facepalm

          Resources for an update bork-checker

          @DaLo: "How many companies, let alone NHS trusts do you know that have the budget and resources to employ an AV update bork-checker?"

          Maintain the one Windows image that's rolled out across the organizatiion and impliment your own patch cycle. Include a roll-back mechanism for patches that bork the system. Test the updates before adding them to the image. Would take about one-man-days work.

        2. Trevor_Pott Gold badge

          Re: What happened...

          Expecting your vendor to do testing for you is socialism. It's nothing more than entitled sysadmins looking for a hand out. This is something everyone should be doing for themselves. If you want to have someone else do testing then you pay them for it. Specifically. Independently of the definitions.

          I can't believe all the crazy socialists around here. This planet sure is going to hell.

      2. Greg D

        Re: What happened...

        On the contrary, this should have been picked up by Kaspersky's internal testing processes. It's not down to the Enterprise sysadmins to test 3rd party products - that's not our job!! If anything, it's easier to test if you work for an SME (5-200 PC's) rather than an Enterprise with 1000+ PC's.

        I work for a company that employs over 10,000 people, with over 15,000 workstations and servers and trust me when I say, we are completely reliant on the 3rd party vendors to do their due diligence before releasing updates to their products. We simply don't have the time or manpower to test every update, that's plain ludicrous!

        Our job is to design, implement and maintain infrastructure. Not test other people's code.

    2. Brian Miller

      Re: What happened...

      You (the system administrator) can't test it before it's applied. (Well, I couldn't do that with McAfee a dozen years back.) The definitions go out automatically, because you'd be testing those definitions every day, and the sales staff are opening dodgy attachments right now.

      The real question for the various AV firms, and they've all been hit with this, is how did it escape their testing??? Shouldn't this stuff be automatic? Shouldn't the testing come up and say, "hey, this borks a normal installation," and raise a big red flag?

      1. Captain Scarlet Silver badge
        Stop

        Re: What happened...

        "(Well, I couldn't do that with McAfee a dozen years back.)"

        Odd could do that around 8 years ago in McAfee's ePolicy program (2.0 I think it was)?

        Several ways to do it but I tended to be lazy so had the IT Department get the first batch as we had a dedicated super agent server, set it up to send out to the other super agents several hours later (If our machine would corrupt jump on server and tell it to revert).

        Not the best way but meant no need to manually test anything or use the BETA Dats from McAfee.

    3. Anonymous Coward
      Anonymous Coward

      Re: What happened...

      Those controls are called "the internal QA process".

      Clearly that's lacking over at Kaspersky and several others. However, that said, there's always the internal process that does everything over and over… until someone pushes the wrong button (instead of 'send back', click on 'release').

  2. Select * From Handle
    Facepalm

    This is why i hate Kaspersky..

    I used to love Kaspersky, i found that on my home computer that it was the bees knees worked flawlessly. When it came to changing the antivirus at work we decided to go with Kaspersky's Enterprise software. For the first year it worked flawless again, with a fairly easy deployment process. But as soon as the Kaspersky client workstation software found that the licence keys were due to "expire in 7 days". It throw a massive hissy fit and locked up all the computers, blocking all network traffic on each machine.... I then had to go and manually remove and re install each antivirus with new keys... (this took a while) now currently into my second year with Kaspersky 3 weeks ago it decided to lock up all the machines again with these false positives.... I have also been told by a friend that he had the same problem upgrading the versions he was deploying. So to conclude Kaspersky appears to have an inbuilt kill switch that will hold you hostage and the trigger is loose, i will not be using next year...

    1. Anonymous Coward
      Anonymous Coward

      Re: This is why i hate Kaspersky..

      Upgrading from regular Kaspersky AV to Kaspersky PURE bricked my home office system, and then they had horrible 1-2 years out-of-date documentation on their self-help website. I don't think I have ever been so mad at a software company.

  3. paulc
    Trollface

    but, but, but...

    "falsely classified a Windows system file, specifically tcpip.sys, as malign and quarantined it"

    that's the best possible way to make a windows box immune to infections... ;)

    1. wowfood

      Re: but, but, but...

      I remember norton doing that to explorer.exe while I was at uni. Not once, not even twice, but three times, several months apart.

    2. Roland6 Silver badge

      Re: but, but, but...

      We are sure that tcpip.sys really was clean?

      Remember from a few years back the MS update to ntfs.sys that caused systems to BSOD, that was blamed on MS, but was subsequently found to have been caused by an undetected NTFS.SYS infection...

      1. Anonymous Coward
        Anonymous Coward

        Re: but, but, but...

        The infection that was pushed out via microsofts update system on behalf of the NSA? Yeah I remember that.

        1. Alistair MacRae

          Re: but, but, but...

          That sounds like the plot to a sci-fi movie

          The AI at Kaspersky's lab was asked the best way to make windows secure.

          Computer thought well all these problems are coming from in via the network adapter. I will prevent it working it in the future.

          Maybe if given longer it'd think, but they come from the internet I must dismantle the internet.

          Longer still and it would realise that people are the problem and get rid of them.

          We got of lightly... this time.

  4. Joe Drunk
    Facepalm

    Nothing new

    This has happened before although incidents haven't been reported as frequently as they used to. The last time I had any AV on my XP machine was when a certain nameless suite deemed WINLOGON.EXE to be malicious and quarantined it. Couldn't un-quarantine it since I was locked out of Windows as a result but was able to manually restore it from Windows install CD - not something a noob would be able to do. I now run naked with no AV or firewall and my Windows 7 system boots in 15 seconds.

    I know many of you here do tech support for friends and family like I do. Notice how much a dead weight any AV is against garbage toolbars, search re-directors (CONDUIT et al) and pop-ups? I clean this crap on a regular basis, being paid in dinner/beer. Would hate to do this for a living.

  5. Anonymous Coward
    Anonymous Coward

    So given that this isn't a central NHS system and isn't enforced by NHS policy what does this have to do with the NHS? Other than as an example of presumably one of many large orgs affected by this?

    1. Badvok

      Probably not 'one of many large orgs', the NHS is probably one of only a few running such an old version of KAV.

  6. Mark #255
    Facepalm

    Why would you standardise on *one* AV solution?

    As the news item proves, if you have an AV monoculture, everything's fine until your entire network gets b0rked by the fuck-up.

    Perhaps the bloke who suggested this could talk to a health professional about analogues in the medical world (MRSA springs to mind), and see what they think about a single defence vector.

    1. Greg D

      Re: Why would you standardise on *one* AV solution?

      Yeah, on your own son! Using multiple software packages to do a single job is insane. Management would be a nightmare, as the NHS guys are finding out.

      So when one of the AV packages b0rks half your computers, and the other half are fine, how are you going to explain that to your staff and managers? Then manage the rollout of the fix without hitting all machines? I could go on but I cant be bothered.

      And trying to draw analogues with the health profession is stupid. If they released a vaccine that kills a man, they'd be sacked and possibly jailed. Not only that, how do you compare a biological vaccine to a software algorithm? You really haven't thought about this have you?

      1. Juillen 1

        Re: Why would you standardise on *one* AV solution?

        Wow.. You think there's only a single antibiotic on the market that everyone uses, because all infections are magically cured by it?

        There are loads of antibiotics, and a whole array of drugs used to treat conditions, largely because there are strains of infection, mutation, slight differences in weaknesses between them and so on. If you simply had one drug for one condition, you'd have a reasonably ineffective drug that would get rid of some conditions (screwed if you weren't one of the ones it treated), and it'd very soon be the factor that every infection mutated to overcome (the big target).

        1. Greg D

          Re: Why would you standardise on *one* AV solution?

          If this was in reply to me, I think you missed my point entirely. The point being, trying to draw analogues between computer AV and medicine is stupid.

  7. Mr_Pitiful

    I thought the NHS / Government Orgs had an 'Airlock' between internal systems & the real internet

    I look after some NHS monitoring systems and they are pretty locked down, no need for AV

    as you can't get to the Internet from them, even though they are visible from the outside for me.

  8. Andy The Hat Silver badge

    Sorry, all this about 'why did the sys admin not test it ..." and all that rubbish ...

    The whole point here is that a basic system file, not even something obscure, was borked. In this case, why didn't the AV vendor apparently test it on a windows box before it was unleashed?

    Where there's blame ... and all that.

    1. Pascal Monett Silver badge

      Indeed, this is far from being the first time such a thing has happened, and far from the first vendor to which it has happened as well.

      It is clear that there is no basic vendor test for an update against standard Windows files. In this particular case, it is equally clear that the vendor did NOT test it before rolling it out, which smacks very much of lax control procedures.

      But given that their LiveDisk was not even able to boot the last time I tried it, I cannot say I am surprised.

      Actually, I am surprised. I'm surprised this kind of thing doesn't happen more often.

      1. Anonymous Coward
        Anonymous Coward

        @ Pascal Monett: "Actually, I am surprised. I'm surprised this kind of thing doesn't happen more often."

        It does, far more often than you think, but some AV vendors employ powerful damage control lobbyists to make sure you don't get to read about it.

  9. M Gale

    Working perfectly.

    It detected W32.NSA.Backdoor and removed it!

  10. Stevie

    Bah!

    Everyone worries that cyberwarriors will bring the country to its knees by using Trojans and viruses, but I can see that the best digital attack in the world would be a variation of this opening salvo by Kaspersky - the sabotaged AV update.

  11. bigfoot780

    OS deployment time

    Good time to rid all those pcs on xp.

  12. Sergey 1

    Using our tax to buy windows?

    Serves them right...

  13. i like crisps
    Facepalm

    The words EGGS and BASKET spring to mind...

    ...That is all.

  14. triggerhappy

    This did not just affect NHS computers. Commercial computers as well, esp those running Win 7 32bit SP1. We won't recommend Kaspersky ever again.

  15. TonyJ

    So to the downvoters of testing...

    Presumably you all just allow Windows Update to run on all of your clients and servers without first testing they don't balls up your systems in some obscure way too?

    AV cock ups, whilst fewer than they used to be, are anything but unheard of.

    Budget? Simple - let your IT team have the releases first. These individuals should have the necessary skills to remediate any damage.

    I find it laughable when people say "we don't have the time, budget or resources" - this doesn't have to be any more onerous than you make it.

    But downvote away.

    1. Anonymous Coward
      Anonymous Coward

      Re: So to the downvoters of testing...

      It's not a windows update!!!!!

      1. TonyJ

        Re: So to the downvoters of testing...

        I didn't say it _was_ a Windows update but I was making a point that I presume all the people who are too time/budget/whatever excuse constrained to test updates that have, in the past, been known to cause issues occasionally also don't bother to test their Windows updates. Or their application updates.

        Yes, for remote working/home working staff it makes sense if you're not using something like an always on VPN/DirectAccess solution to allow their systems to autoupdate and then there are times where you have to weigh up the impact of a zero day exploit / piece of malware etc and the damage it would cause over the potential instability a remediating patch or update would potentially cause if it's installed, but most of the time there is zero excuse for what is effectively laziness.

        And yes I agree that the vendor should have caught it but they didn't and the IT staff should have therefore had policies in place that caught it before it became such a huge issue.

        Christ - would you just roll out a brand new application suite without any testing?

        1. Juillen 1

          Re: So to the downvoters of testing...

          You really don't work in that kind of environment do you? The way this works in enterprise is that they run very light teams on operations that keep the servers running, monitor and tweak the systems that host hundreds, if not thousands, of applications, check logs for security intrustions, upcoming errors, run diagnostics, herd requested changes through the process and so on.

          Where you have a rapidly changing application (such as anti-virus, where if you don't have the absolute to the minute update and you get compromised by lack of it, you can lose your job), there is a balancing of risk very much in favour of trusting the third party. And you have a contract that hurts them if they take down your infrastructure by oversights such as this (and 'insures' you for any damages etc.). Contrary to what you seem to believe, there isn't infinite resource to check absolutely everything all the time, and if you try, it's a long, long queue. The job is managing risk, and in this case, it's the risk of being exposed to a virus infection that may leak your sensitive data to the world (big, big risk, and a fair likelihood), or the major vendor fails to test correctly to the level that it damages your infrastructure.

          New application suite and antivirus definition update are two completely different things.

  16. Anonymous Coward
    Anonymous Coward

    Perhaps you should release updates to all computer names ending with a 1, wait a while, then release to the rest.

    Then after a month, make it 2's first.

    If the names are randomly allocated, it should give a good spread across functions.

    1. Jamie Jones Silver badge

      Wow! Rolling/staggered updates!

      Why didn't anyone think of that before(!)

  17. Anonymous Coward
    Anonymous Coward

    Microsoft malign tcpip.sys ..

    "A Kaspersky update released on 25 October falsely classified a Windows system file, specifically tcpip.sys, as malign and quarantined it."

    Well, tcpip.sys is malign as it allows Windows to connect to the Internet. [downvote -99]

  18. Alistair MacRae

    What happend to you Kaspersky?

    You used to be cool!

  19. Mark C 2
    Megaphone

    NHS

    "on the grounds of consistency, easing central support, and economies of scale ... but to my knowledge that never came to fruition"

    Says it all about the Public Sector really. They have an unlimited supply of tax-payer funding with no ownership so why would they try and become efficient?

    I think I could save the UK £1 Billion pounds on my first day at work if I was in charge of the NHS:

    - Tender for a UK-wide public sector mobile phone contract with one supplier, 2 basic handset to choose from

    - Tender for a UK-wide contract for all NHS consumables (bandages, syringes, saline, etc)

    - No First / Business class travel (all of the public sector)

    - No bonuses (all of the public sector)

    - Create a standard desktop build for the public sector (yes, there will be exceptions)

    - Use open source software

    - Secondments from Private Sector to the Public Sector (to change the culture)

    - Build a Premier Inn style hotel in London and make MPs / Lords live in it (with suitable security)

    Please feel free to add to the list...

    1. Anonymous Coward
      Anonymous Coward

      Re: NHS

      My wife's NHS mobile phone is a Vodafone Blackberry Curve - everybody else I know who has a NHS business phone has a Blackberry Curve with Vodafone

      Re: UK tendering - given the size of some of the NHS Foundation Trusts - who have their own budgets - these would have to be broken up - that of course will happen for free...

      When I worked for the UK MOD I never travelled Business/1st class - I don't know anybody who ever did.

      Re: No bonuses - yay - way to go - no incentive to try very hard - just do enough not to get fired...

      Re: Standard desktop build - so who would get the Monopoly to provide every single PC to the whole of the Public Sector? HP, Dell, etc? How many machines would that be - how many perfectly good PCs would need to be dumped and replaced right now - who would decide on the minimum spec? You? Apparently (and what would the minimum upgrade time be? Or what if that manufacturer stopped making that PC? And because it's a standard build all those PCs would need rolling out across the whole of the Public Sector at the same time? Or if you phased it in - over how long? How long would the Public Sector have multiple classes of machines to support?)

      Re: Open Source Software - brilliant idea - retraining everybody to use Linux instead of Windows, LibreOffice instead M$ Office, etc will happen by magic, be completely free and not disrupt the workings of the NHS system at all (because the manufacturers of Digital X-ray machines, etc will gladly and happily convert all of their software to work on Linux based OSes instead WIndows based ones)

      Re: Secondments to/from Public/Private sector already happen - most senior managers across the Public Sector are brought in from successful Private Industries (of course that'll stop when you kill off the bonus system)

      Re: MPs/Lords - OK, apart from the fact they're not "Public Sector"...

      1. Tyrion
        Stop

        Re: NHS

        >> Re: Open Source Software - brilliant idea - retraining everybody to use Linux instead of Windows,

        That's the typical F.U.D M$ and their dependents use to justify spending a fortune of tax payer money on software licences every year. If you go from 95 -> XP users need retraining, if you go from XP -> Vista->7->8 (especially 8), users need retraining. Hell, every iteration of M$ Office requires retraining. Retraining is a constant, and therefore it's a blatant lie to suggest it's more costly to move to Linux.

        There are many examples of successful migrations to Linux and the subsequent long term savings. The cost of dealing with Anti-virus, Anti-malware, anti-world&dog alone makes Windows incredibly expensive to maintain.

        When I hear about incidents like this I just laugh. Windows is disaster waiting to happen. I've seen it so many times, and yet the UK government loves to throw tax payer money at Micro$haft for the privilege of being $hafted. But this isn't anything new. Governments around the world waste vast sums of money on useless things. Procurement is rife with corruption and overpayment.

        >>LibreOffice instead M$ Office, etc will happen by magic, be completely free and not disrupt the workings of >>the NHS system at all.

        And events like this misbehaving antivirus software don't disrupt the NHS? Heaven forbid they actually get infected with a virus or rootkit. On Linux, the machines wouldn't even need to run an AV. That alone would save a fortune in maintenance and licensing.

        1. Juillen 1

          Re: NHS

          *Cough* You work in the NHS in ops or IT service delivery do you? I thought not..

          If there's the slightest change that isn't agreed by the Unions (and Linux is a big change for the whole suite of software that's required and available), the Unions can, and will invoke walkouts or strikes if necessary. There's a hell of a lot of negotiation involved.

          From XP -> Win7 isn't necessarily a retrain. There's the familiar start button kinda thing, and most icons on the desktop. Win7 to Win 8 is. You'll note, there isn't much windows 8 in industry.

          Office XP to Office 2007 was a retraining. 2007 to 2010, not so much. 2010 to Libre Office would be a definite.

          Really, after working in big companies on international IT projects, and working inside the NHS in IT ops, running my own show, and consulting with various companies, I hear the kind of rant you're coming out with, and it's almost invariably from people who have never run heterogenous systems, never done major migrations, and never had to be involved in the whole business aspect. The "Well, we can rip everything out over a few weeks, and put it back so the lights look the same, and nobody will notice" plan is something I've seen time and time again. And it leaves a lot of extremely unhappy users, failed services, and project planners kicked out on their derrieres for incompetence.

          Oh, and Linux boxes do run antivirus. For a good reason.

  20. Captain Scarlet Silver badge
    Paris Hilton

    Version 6?

    I might be confusing a home version but I thought Kaspersky Anti-Virus was way past version 6 (Googled it and everything), if it is a very old release thats a lotta back releases they are supporting

    Paris just in case I am being silly.

  21. Chris Evans

    ...apologise for THE' inconvenience. NOT 'ANY' inconvenience

    That should be "Kaspersky Lab would like to apologise for THE inconvenience caused.".

    Companies saying 'ANY really annoys me!.

  22. Anonymous Coward
    Anonymous Coward

    ...No First/business class travel ever...

    Except where it is cheaper than 2nd class, right, because we want to save money? A couple of years ago my County Council employed colleagues and I needed to go to London. County policy was we had to travel by train. We did some shopping around, and found first class return tickets for about £86 each. County policy was no first class travel, so we got told we had to travel second class. At around £150 each. For 10 of us. Common sense not required thank you!

This topic is closed for new posts.