By that I mean what the fuck.
Patching and keeping track of patch releases is a daily house keeping task that every sys admin must do to be worthy of the title.
Roughly a fifth of the public-facing Citrix devices vulnerable to the CVE-2019-19781 remote-hijacking flaw, aka Shitrix, remain unpatched and open to remote attack. Positive Technologies today estimated that thousands of companies remain open to the takeover vulnerability in Citrix ADC and Gateway. A successful exploit would …
It's quite possible that a lot of these are being used by companies who don't have what you and I would recognise as a sysadmin. Small businesses often run with no IT staff at all; somebody, if they're lucky, changes the backup media and for anything else they call in a specialist as and when they need it. On occasion I have been that specialist. Such firms won't even know that this is an occasion when they need to call someone in.
This raises a question. Given that vulnerable, exposed systems can be located from the net should vendors undertake such searches and patch them remotely?
Well I think it's obvious vendors would never reach into a customer's site and patch themselves without the customer agreeing to it in advance.
For a serious issue the most they could do is release the fix to anyone who has a valid serial number regardless of support status(or end of life status). Even if the fix came with disclaimers that it hasn't been tested on a given platform (but "should" work), unless the platform is so old that it can't work (in Netscaler's case I would assume this would be really old perhaps 32-bit platforms or before they went multi core("nc" series) not sure when that was, before I was a customer anyway which was late 2011).
Last year I tried to get some newer code for Cisco ASAs that a previous network engineer thought there was no need for support on since they "hardly ever fail", hadn't had a patch in 4-5 years(I checked, more than 100 security advisories released). Unfortunately they were end of life and while I did not need "support", I just wanted to download the code that was there already (and be happy to pay for it), Cisco said no. Fortunately was able to replace the ASAs with another technology product not too long after.
There's also the thing where the device 'works' just fine until an upgrade is applied to it, at which point the entire thing breaks and has to be rebuilt.
We had to do that to our netscalers at least twice. When we found out about this exploit, the devices had been removed from active use but essentially running idle for a while; our fix was to shut the damn things off permanently.
curious what kind of issue came up that had to have things be rebuilt? I wouldn't consider myself to be a Netscaler expert but have managed netscalers for web and mysql traffic since 2012 (before that I used F5 for many years). Code roll back should be very simple. (since I have HA, I do all of my upgrades via CLI, haven't tried the GUI to upgrade since I setup my first netscaler).
I was on the 9.3 code base for many years well past end of support. In the early days of using it seems every new release broke something minor, fortunately I don't recall anything ever critical being broken(enough to want to roll back anyway if my memory is right at least). I did hold off on upgrading to 11.x from 9.x due to a mysql query routing bug that took about 2 years to track down and get resolved. I skipped 10.x entirely, straight to 11.0 now 11.1, no need to go higher as long as 11.1 is supported. Though haven't run into any upgrade issues in probably 3-4 years now.
I have not used them for anything like RDP or virtual desktops or anything, strictly http/https, dns, layer 4 stuff, NAT, mysql load balancing, and VPN(only one in the company that still uses citrix vpn, everyone else uses pulse secure). Originally went for Citrix instead of F5 because I was curious about the integrated VPN, the mysql load balancing(neither of which F5 had at the time build into LTM anyway, F5 firepass was a thing but not part of the load balancer), and I had heard good things about Netscaler in general so wanted to give it a try.
Most annoying issues with netscaler over the years for me anyway was that mysql query issue("the NS is not forwarding the "PREPARE statement" which was sent after the "CLOSE statement "by the client."), and problems with the Mac Access gateway VPN client. Got pulse secure after at least 18 months of support on the mac vpn client that we finally discovered the design flaws in the client side that really wouldn't get fixed anytime soon(if ever). Worked fine from windows though.
Had to replace 2 netscaler SSDs in the past year that was a bit annoying as well, the spinning disks in the older platform continue to work fine after almost 8 years.
There was a major change in 11.0 code (I think) for how Citrix sessions were routed that required an undocumented configuration change that may have been the issue - if you upgraded from 10.x to 11.x and hit it you needed to rebuild as you had no rollback path that didn't leave you with broken system.
The fix was to call Citrix to get the undocumented fix and later software releases implemented the fix as part of the upgrade process.
These things happen...unfortunately there are multiple ways of configuring devices and test every possible combination is difficult.
If you don't have a support contract you can't get the fix. Also increased possibility if you don't have a support contract Citrix won't know to try to contact you about the issue. I bet hundreds of those devices at least are also end of life so probably can't get a support contract even if they wanted one(I have a pair of fully functional 7500s that are well past EOL though I'm sure will happily run the latest 11.1 code base at least they have been retired for 18 months I couldn't find a good purpose for them). Those 7500s were purchased in 2011 and literally wouldn't break a sweat running full production load in 2020(the current Netscalers I have run memory and cpu average 5% - I'd wager the 7500s would run the same traffic at 10-20% load tops).
For netscalers at least the code really isn't platform dependent so if you have a support contract for some other netscaler then you can download any version for any platform.
But I'm sure lots of folks out there have these appliances who don't have support, and some subset of them probably don't care enough to do anything about that.
If you don't have a support contract you can't get the fix.
There are two fixes.
One is a configuration change which mitigates against the attack. It's available on Citrix's public internet, and is just a few commands to slap into the console. It's basically a rule that seems to trap what the attacker is trying to do and returns a specified response.
The other is an actual fix to stop people going 'https://the.device.com/../../scripts/breakme.sh' and binning any attempts to access parent folders in URLs. It's an actual patch which makes the folder handling more sensible and secure, but that one is an on-support job.
So you can protect yourself even without active support.
Biting the hand that feeds IT © 1998–2020