back to article A third of London boroughs 'fess to running unsupported server software

A third of London councils and more than a quarter of England's metropolitan authorities have admitted to using unsupported server software – and three are still running Windows Server 2000. The figures were revealed under a Freedom of Information request that asked the councils which of the following they ran anywhere in …

  1. g00ner

    Server 2005???

    What's Windows Server 2005? Must have missed that release :)

    1. Sgt_Oddball Silver badge

      Re: Server 2005???

      Got back and re read. Notice those extra 3 letters. SQL. So it's not the OS server software but the database package instead.

      (SQL 2014+ is worth the upgrade costs alone for the awesome sauce it being for competent DBAs).

      1. Chika

        Re: Server 2005???

        I'm looking at the paragraph right now. "Windows Server 2000, 2003 and 2005; and Microsoft SQL Server 2005 and 2008."

        I never heard of Windows Server 2005 either.

      2. g00ner

        Re: Server 2005???

        Perhaps you need to go back and re-read it :)

        Will accept your apology in advance

        1. John70

          Re: Server 2005???

          Windows Server 2000, 2003 and 2005; and Microsoft SQL Server 2005 and 2008.

          It does say Windows Server 2005 as well as SQL Server 2005.

    2. kain preacher Silver badge

      Re: Server 2005???

      THat's cause it was for the mac only . Like Office 2004

  2. Doctor Syntax Silver badge

    Isn't it odd that when it's suggested that they run Linux or a BSD people will come out of the woodwork and claim it can't be supported?

    1. Korev Silver badge
      Linux

      If you've got a team of Windows server admins than that might be the case. Also, for "hosting" some applications the vendor dictates the OS, DB etc.

      Running Linux etc isn't always the answer here, I'm sure an equivalent survey would find places still on Solaris 9, RHEL4 etc

      1. m0rt Silver badge

        I think we need a new OS with a completely different philosophy in design and roadmap.

        We have become addicted to upgrades, fast implementation and buggy code.

        Or just use openBSD.

        1. AMBxx Silver badge

          You'll probably find that a lot of these outdated versions of Windows are being used because they're running really old software that can't be upgraded and isn't compatible with more recent versions.

          Think of all the software companies that used to exist (plus contractors) in 2000-2005. Some of that software will still be important and can't be replaced wihtout significant cost.

          1. Boothy

            I can remember in around 2002, we were looking at upgrading our then aging UNIX platforms. We were using AIX back then, with most systems running 4.1.* or 4.2.* so were all years out of date (some quite a lot of years!).

            The plan was to shift everything to 5.* which also including new hardware in many places. (We did no Open Source at the time, so Linux wasn't even discussed)

            One major bit of critical licensed software we were using (around £200k per year for a single server licence), turned out, that even the latest releases from that year (2002), were only certified for AIX 4.2.1 & 4.3.1 (1997 and 1998 respectively). it hadn't even been certified for the newer point releases for 4.3.x, let alone 5.n!

            In the end we just took the risk and tested the software ourselves. We told the vendor that if they refused to still provide support, we'd go elsewhere for a new product.

            About 2 years later, they still hadn't certified for AIX 5, and as we were planning on a major update to the platform, we just went elsewhere (it was a data integration and transformation engine). We also told the vendor specifically why we were not renewing with them.

          2. John Brown (no body) Silver badge

            "You'll probably find that a lot of these outdated versions of Windows are being used because they're running really old software that can't be upgraded and isn't compatible with more recent versions."

            I once did a fix on a PC running Windows NT4 at a very specific patch level for Rolls Royce. It was one of a pair running a pair of specialist machines for finishing jet engine parts. The machine control s/w would ONLY run at that patch level, nothing higher, nothing lower. Some software is just very poorly written or takes specific advantage of buggy or transient "features" so it's no surprise that large organisations will run ancient kit to support some apps or hardware that needs to be retained for whatever reason. In the case of councils, as others have mentioned, there will be legal requirements to retain access to some systems or data that can't be economically moved to newer systems, especially as central government contuies to squeeze there budgets.

            Considering the recent headlines over Northhamptons budgets, I'm surprised they didn't get an honourable mention in the list.

          3. GruntyMcPugh Silver badge

            @AMBxx

            Indeed, I work for local Govt these days, and the sheer number of applications we support is bewildering. When I worked for an ISP, we had 'CableMaster' the app that managed customer accounts, and sent signals out to (then analog) set top boxes to provision their packages, some telephony management intermediate software, file and print, email, and the usual HR / Payroll. That was it. In local govt, we do a lot of things. We have a database covering every tree along every road and pathway for instance, which we inspect. A database for road maintenance, we run leisure centres, recyling centres, we register births, deaths, and marriages, licensing for boozers, housing, benefits, planning applications, museums and galleries, social care, education, libraries, ... so some of these apps just do what is needed, and sometimes the software vendor shows very little interest in upgrading the app for a handful of clients,.... they'll charge for changes, but that's about it. Therefore moving platforms just isn't possible often.

        2. BinkyTheMagicPaperclip Silver badge

          I'm not sure if suggesting OpenBSD is sarcasm m0rt, but if it isn't you clearly haven't used it much.

          I am an OpenBSD fan and use it frequently on the desktop as well as infrastructure, but the support policy is *one year* - current and previous release, new version approximately every six months. There are no LTS versions.

          It's true that OpenBSD is secure, and that there have only been two remote holes in the default install, but the default install is mostly limited to OpenBSD specific infrastructure software (firewall, email, very basic web server, routing, dns). If there is no local execution of programs or third party software then yes, upgrading is less essential, but many people need additional functionality.

          If there is a need to go beyond the default install ports/packages are not audited to the same extent as the base install, there is no binary compatibility, and the ports tree is a moving target, so a couple of releases on it's possible it will not build against an unsupported release.

          Furthermore, OpenBSD's policy is pretty much 'security before all else' - firewire, bluetooth, and the Linux compatibility layer were dropped completely because they weren't being adequately maintained and security couldn't be guaranteed. Currently hyperthreading is disabled by default under OpenBSD due to the speculative execution information leakage issues, which is certainly an effective mitigation but (on Linux) leads to around a 30% performance degradation in some scenarios.

          If a council can't upgrade beyond a Windows Server version released in 2000, I'm not sure BSD is really the best idea..

    2. big_D Silver badge

      Using Linux or BSD doesn't change anything. It is the attitude behind such decisions that needs to change.

      I know one company that was still pushing out servers with SUSE Linux 7 on new machines to their customers in 2015! Version 7 was released in 2000! They only switched to a newer OS, because the new RAID controllers couldn't be used.

      The attitude was, "its Linux, it doesn't need patching".

      That said, I've also seen systems on Windows 2000 hanging around for compliance reasons. So, with a 10 year storage requirement for historical data, I can imagine that 2020 is about right for replacing Windows 2000, if support ended in 2010. If it is an old system that is no longer actively used, you have to either keep it around, spend big bucks on a newer version, just for it to sit in a corner and gather dust, "just in case" there is a problem with an old customer or spend big bucks to have the historical data transferred to a new system, where it just clogs it up.

      A lot of people will settle for the "risk" of having that old kit around, "just in case", rather than investing heavily for no gain. If those systems are then air-gapped from the rest of the network, many can "live with it".*

      * I'm not condoning the practice, just saying what I've seen happen.

  3. Anonymous Coward
    Anonymous Coward

    Universities

    Wait until they FOI universities, I'd wager it'll be 90%+ running unsupported OS flavours

    1. sal II

      Re: Universities

      Can't comment for all, but I have worked in IT of 2 major London Universities and both were the exact opposite - always pushing for the latest versions as soon as they come out.

      Sure they had some other "skeletons in the closet", but keeping old OSes around wasn't one of them.

      1. GruntyMcPugh Silver badge

        Re: Universities

        @Sal II

        My experience too,... I've worked for two different Universities (well, one was a Poly when I first worked there) and we were early adopters, at one point we had the largest deployment of Microsoft Exchange in Europe (so the guy from MS told me at the SIG meetings we used to attend), that was Exchange 4.0. (there was a DOS client back then,......)

        At another Uni (a real one this time) we were quick to adopt Linux, had a web server when the web was really just academics, and made some of our data publicly available via the web (I won't bore you with the detail, but the guy that came up with the idea had a light bulb moment down the pub, while looking at their CD Jukebox)

    2. BigSLitleP

      Re: Universities

      Agree with above comment. I worked for a major university in Birmingham and they were always pushing the bleeding edge for their equipment.

    3. John Brown (no body) Silver badge

      Re: Universities

      "Wait until they FOI universities, I'd wager it'll be 90%+ running unsupported OS flavours"

      Most of them very probably do have at least some legacy kit for specific reasons. I deal with a number universities and as others have just said, universities are likely to be at the leading edge if not actually bleeding edge for most of their kit. (one Uni I did some work at had just completed a complete server room refresh and I notices a beige box in the corner, still powered up. I asked what it was. It was a Windows NT Server box with a multiplex modem in it for those professors who still insisted they needed dial-up access when off on expeditions, sabbaticals etc. (this was about 5 years ago, so hopefully that's now gone to the bit bucket in the sky!)

  4. big_D Silver badge
    Coat

    Probably good...

    that they didn't ask if they were still running Windows NT Server! :-D

    1. Anonymous Coward
      Anonymous Coward

      Re: Probably good...

      You'll be surprised - at one of our toll road plazas I saw Windows NT 4 Workstation still in active use a couple of years ago...

      The problem was the hardware - eventually NT4 was forced out in favour of Windows7...

    2. kain preacher Silver badge
      Coat

      Re: Probably good...

      Or Netware. Gets my coat. It's the one that has the Windows for Workgroups 3.11 disks in it.

  5. Anonymous Coward
    Anonymous Coward

    They'll be having fun with the PSN accreditors for the next 12 months.

  6. Sixtysix
    Flame

    Lies, damn lies and FOI

    We ran a server 2003 instance until very recently, and I constantly got criticised for the "gross security risk" that represented.

    This is WRONG for *some* use cases.

    On a well designed infrastructure, it is more than possible to design the network operations in such a way that an older, but still critical, application can run on unsupported Hardware/OS/Application framework and etc. safely - if it is only used internally, and cannot reach/see the internet.

    It takes effort and planning to ensure that it cannot be reached except as required to provide the "service" it exists to provide, and is only accessible by the clients and methods essential to that service... but that's why internal DNS, subnetting, VLANS, Reverse Proxies and Firewalls exist: to mitigate, control and contain risk.

    So MUCH of my staff's time is wasted responding to FOI requests that are just used to sell my details to marketing droids... that I don't want to hear from (and no I don't want your white paper, didn't give you permission to store my details, so GDPR them off your contacts system, please, thank you and goodbye).

    1. sal II

      Re: Lies, damn lies and FOI

      You know that there is something called operational risk, in addition to the security risk.

      Your box can be firewalled to death or completely cut off from the network, it won't help you if something f**ks up with the OS or the DB and you are left on the mercy of "best effort" support by MS.

      What irks me the most is that such systems are dubbed "critical" so can't be migrated, yet there are rarely provisions for what happens if they just die one day.

      1. dansbar

        Re: Lies, damn lies and FOI

        In my career, spanning Windows NT4 through 2016, I can remember all three times that I have actually approached Microsoft for support and on all occasions it was in the first year of release of the OS. There are so few issues that actually require escalation to Microsoft that cannot be dealt with by a competent support team.

        If your database fluffs up, Microsoft couldn't care less unless it's a bug in their software. You probably would have found that diring the first 5 years of running the database on the platform.

        Running on out of vendor support OS is completely safe if the network is correctly configured to isolate the OS and only permit access to the application. No internet access from/to the server blah blah. In fact, that 15 year old environment not get patched, poked and prodded all the time by external influences is likely to be rock solid.

  7. Ken Hagan Gold badge

    Can we trust the answers?

    How many of those questioned actually know that Server 2008 is different from Server 2008 R2?

    I bet some don't. I bet some respondents are just PR staff fielding FoI requests to the best of their knowledge, which in this case is flaky.

    Mind you, the "R2" suffix was always a really crap product name. Might as well have called it "Sorry for that turd. Is this one any better?".

    1. Roland6 Silver badge

      Re: Can we trust the answers?

      Also I expect the survey didn't ask what proportion of their business systems are running legacy OS's and the importance of such systems to the business.

      I have a WfWG laptop that gets powered up occasionally - when I have need to go that far back; I also have a laptop running XP with a suite of useful tools; however, all my everyday business systems are running OS's that are in support and thus receiving security updates.

  8. Domquark

    It's all about the money...

    We have all seen it and been there - a company/organisation that you work for is running outdated and obsolete kit, desperately in need of an upgrade. However the management won't listen to you, don't believe that IT systems are important, and use any company cash for a nice pay rise and a final salary pension.

    One company (that I know of) in the Temple (in London, an area for Solicitors and Barristers) is still running Exchange 2003!

    1. John Doe 6

      Re: It's all about the money...

      ...if they are unwilling to spend money on software they should run Gentoo or Debian.

    2. Anonymous Coward
      Anonymous Coward

      Re: It's all about the money...

      Could be worse, when I moved to one company thinking I could move beyond creaky 2003, I found out they were still using ES5.5. Fortunately now on hosted Exchange, and it works fine.

      Not that I can speak, one customer is still running ES2K3 because the gateway (written by me) isn't being updated. Blame Microsoft for that, the interface changes after ES2K3, the customer is unlikely to pay for a gateway upgrade, and there are no economies of scale to fund ongoing development (it's a very specialised gateway used by a tiny number of customers).

  9. Anonymous Coward
    Anonymous Coward

    Potentially not THAT bad

    If they're running in isolation, locked down, maybe just for compatibility with an old application then that seems pretty reasonable.

    I had to repair a system running Windows 98 a few months ago. An embedded system that's been running every day for 20 years (rebooted daily, mind- it's still Win98) and shows no sign of failing, except for the date/time backup battery being too old to work (it was a soldered-on model, it's been out of service for about 10 years but they decided to renew it). I've no significant worries about that system outlasting the rest of the system it's a part of.

    Why upgrade if it's not broken and there's no benefit?

  10. Anonymous Coward
    Anonymous Coward

    Living in the Rotherham council area, around here Windows 2000 is cutting, almost bleeding edge.

    1. John Brown (no body) Silver badge

      ...I thought round those parts, it was still all "them black magic electrickery things that steal your soul"

      1. Intractable Potsherd

        I still have friends and family there (I escaped!), and your characterisation is remarkably accurate, Mr Brown!

  11. localzuk

    Somerset?

    I know for a fact they run more modern OS's than 2003, at least in the Education part. So, not sure that part is accurate.

  12. Spanners Silver badge
    Flame

    The FoI – a glorified sales pitch

    I have come across quite a few of these. You get a request that is a thinly disguised pitch for information so that someone can try and sell you stuff that you clearly aren't looking for.

    This gets the whole idea of the FOI Act a bad name. That beloved public figure Tony Blair allegedly said that passing this was the worst thing he did whilst in power. I am not convinced of that but it is certainly used for purposes very different from what were originally thought of for it.

    1. Anonymous Coward
      Anonymous Coward

      Re: The FoI – a glorified sales pitch

      I remember working in the public sector and being cold-emailed by an events company. Ever dealt with events companies? They're about one step up from recruiters most of the time. Anyway they wanted a copy of our org chart, probably so they could add a few senior names to their cold-calling rota.

      They also made threatening noises about invoking an FOI request if they didn't get the info. I assume this was because they thought public sector bods would fall over themselves to be helpful at the merest hint of being slapped with an FOI request.

      Thing is, this info wasn't that easy to get hold of, I had better things to do like read El Reg, and I'd taken a pretty instant dislike to this company. So I just replied with the contact details of our FOI team and, after another couple of huffy emails, the events company buggered off. As far as I know they never submitted a formal request.

      1. Ken Hagan Gold badge

        Re: The FoI – a glorified sales pitch

        "Anyway they wanted a copy of our org chart, probably so they could add a few senior names to their cold-calling rota."

        These days I guess you could respond to the FoI request with an org chart that was topologically correct but that omitted the actual names for GDPR reasons.

  13. JimC Silver badge

    The FoI – a glorified sales pitch –

    Exactly. How much time would you want your council's staff to take off doing something useful like supporting the end users to respond to something like that. 5 minutes? One minute?

  14. Peter2 Silver badge

    Oh, how terrible.

    Actually, an admission; If I was answering a survey similar to this honestly then i'd have to reply that i'm still running either NT4 or Win2k. (not actually sure which) Additionally, i'd have to admit that it's never had a security patch since being installed and that I have no plans to touch it.

    This is because it runs the firms voicemail system, and came with the telephone system a very long time ago and has quietly kept ticking on since. It's connection to the outside world is via a bank of 56k modems, which receive telephone calls and also do the usual voicemail playback stuff for internal staff. It doesn't even have a network card, being of the vintage where motherboards left network and USB connectivity to be provided by PCI cards, rather than being baked into the motherboard.

    The only way of getting information out of it would be direct physical access to the console (bringing your own PS2 mouse & keyboard + DSub monitor) and then writing something to transfer the data via the serial port. It's sort of more "no risk" than low risk when you consider remote data compromise. It's (still!) got an external support contract for BCM, which ends any concern about it still being kicking around after what now must be about 20 years.

    I have yet to speak to somebody else at an industry event who won't admit to having something really old like this sitting around somewhere.

    1. Wensleydale Cheese Silver badge

      "The only way of getting information out of it would be direct physical access to the console (bringing your own PS2 mouse & keyboard + DSub monitor) and then writing something to transfer the data via the serial port. "

      Ooh, Kermit!

      (It must be knocking on for 20 years since I used that utility.)

  15. Anonymous Coward
    Anonymous Coward

    Heh

    I once did some contracting work for a 'Major Financial Institution'. They were running NT4 servers and a mix of Win 98 and Win 98 SE desktops. This was in 2007-8. They were thinking about moving to Win2k servers and either WinME or Win2k desktops 'within 18-24 months'. Maybe. Win2k3 and XP were not even considered. Macs were 'too expensive'. Linux did not exist, so far as they were concerned.

    Thy did hot-desking. The hard drives in their desktops were, umm, inadequate. This meant that a common problem was that users could not log in, because there wasn't enough space available to store their profiles. You read that correctly. The admins would log in via an local admin on a machine, clear the accumulated profiles, and move on. They spent a significant portion of each day doing this.

    I recommended Win Server 2003 or 2008, and Win XP or Vista clients. This would have meant a complete replacement of their kit, as the Win 98 desktops could not run XP or Vista, and the servers were barely up to NT4. Some still had SP 3 installed because they 'could not run' SP 4. None had SP 5 or later.

    After management recovered from the heart attacks produced by reading how much this would cost, they thanked me politely, paid my final fee, and updated to Server 2k and Win ME before the end of 2008. Exactly where they got the licenses and how they managed to get the things to install is unknown to me. They were still using 2k and ME at least until 2014.

  16. RobertLongshaft

    How many business do you think genuinely are not running unsupported software?

    I'll got for less than 1% and of those that are "clean" 100% of them will be start ups.

  17. StuntMisanthrope Bronze badge

    I, Kerberos

    Let’s just add it to the directory of fail. I’m about to table the opposite approach and structure a whitelist of success within the realm. Dapper threads all day long. #itsaminoritycauseofpublicity

  18. Anonymous Coward
    Anonymous Coward

    How can they still be PSN certified?

    I'm an ex local gov employee (outside of London, but not too far) and worked as an ICT manager and know that most LAs have to comply with the PSN Code of Connection (https://www.gov.uk/government/publications/psn-code-of-connection-coco). Running these unsupported servers is a direct violation of this agreement...so the question is, how can they still be allowed to operate on the PSN network (which handles sensitive DWP data amongst other things)?

    I know where I worked, we had 2x 2003 servers still running and we were allowed to keep operating with W2003 as long as we could show a clear path to migrating off of the unsupported software. However, this is farcical as this remained the case for at least 3 years in a row (and was still the case in April this year) and basically showed that the PSN CoCo is toothless and LAs are getting away with risking ALL of our data!

    This just goes to show how lax central gov is when it comes to this compliance and also how underfunded LAs are. In our situation, it wasn't because we wanted to remain on legacy software, we just couldn't afford to replace the systems running it, but where the MoD have the luxury of paying for extended support, LAs can't afford to do this.

    Come on Central Gov...wake up and smell the sh** that LAs are coping with!

  19. steviebuk Silver badge

    So did you say this..

    ...was just a misuse of the FOI system by Comparex so they could advertise their bollocks?

  20. Daniel von Asmuth Bronze badge
    Windows

    Science Museum

    Once upon a time I went to London and visited the Science Museum. They had a great many legacy technology on display, including Sptifire and Hurricane fighters, which makes me wonder if all those exhibits still enjoy regular support and maintenance from their original factories.

    1. Roland6 Silver badge

      Re: Science Museum

      I wonder how the National Computer Museum at Bletchley Park would respond to one of these FoI requests and what claims the requesting party would make arising from the answers...

    2. Peter2 Silver badge

      Re: Science Museum

      Once upon a time I went to London and visited the Science Museum. They had a great many legacy technology on display, including Sptifire and Hurricane fighters, which makes me wonder if all those exhibits still enjoy regular support and maintenance from their original factories.

      . . . Actually, yes. They do.

      In 1963, a Mr Bill Lear Jr was living in Geneva, Switzerland and flying a surplus P-51. After numerous problems with the starter clutch on his Packard-built Merlin, he contacted Rolls-Royce. They instructed Lear to send them the clutch, which was quickly repaired and returned. Lear adds:

      “I called my benefactor to thank him and to ask him when to expect an invoice. His reply was: ‘My dear Mr. Lear, Rolls-Royce-designed products do not fail. They may require occasional adjustment, but this is covered by our unlimited warranty. So there is no charge, sir.’

      I was blown away. The engine and clutch had been manufactured under license in the U.S.A. by Packard in 1944, yet Rolls still stood behind them in 1963!”

      Apparently this has been found quite handy by the RAF with the Battle of Britain Memorial Flight, but I suspect that the science museum doesn't really require that much in the way of support.

  21. Duffaboy

    Xp is still alive

    worked on a couple of boxes this week

    1. Loud Speaker Bronze badge

      Re: Xp is still alive

      I have an embroidery machine that needs XP. The manufacturer's suggested upgrade is to buy a new £5,000 machine. I met their UK MD recently, and explained their competitors support Linux, and IF I replace the machine, I will NEVER buy into Windows dependence again. I have learned my lesson.

  22. Uberior

    All this really tells us is that we have far too many local authorities.

  23. Anonymous Coward
    Anonymous Coward

    Ooo, I could tell you some stories from a local council Win7 roll-out some years ago.

    "Here's three disks with AutoCAD 2013, we need it installed on those 20 machines."

    "Corporate policy disables all external ports - what do you mean, vehicle testing needs to plug test equipment in?"

    "My machine was re-imaged overnight, where's all my data gone?"

  24. //DLBL SYSRES

    We kept a 2000 server alive until this year solely for the free TS licences that were transferable to Windows 2003. Saved a few quid, now dead,

  25. Anonymous Coward
    Anonymous Coward

    And Sheffield's IT is outsourced to...

    You guessed it, Crapita IT Services

    Not sure about the others

  26. Anonymous Coward
    Anonymous Coward

    Final salary pensions

    Well after putting money aside for final salary pensions, what would you expect.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019