You call it a botched patch, Microsoft call it virtual air gapping.
Patch LOSE-day: Microsoft secures servers of the world. By disconnecting them
Microsoft’s Tuesday patch-fest may have reacted quite negatively with Windows Server 2008 R2 running VMware, leaving servers offline and administrators scrambling to recover IP addresses. Twitter user @Sikorsky78 noted the problem just after the patches hit: It seems that the latest MS patches ate my vmxnet3 adapters on …
COMMENTS
-
-
Thursday 15th March 2018 13:33 GMT Lee D
Bigger question - why are you using static IPs on regular PC's?
To be honest, even servers don't need a static IP. The only exceptions are literally the DNS server IPs that you plug into clients via DHCP or manually. That's it. Even the OTHER DNS servers can be DHCP and picked up from the original DNS server and relayed.
But a client PC? Stop it. Use the computer name. Whack the lease-life up on DHCP if you have to, but static IPs are just silly.
-
-
Friday 16th March 2018 18:39 GMT SImon Hobson
Bigger question - why are you using static IPs on regular PC's?
Because sometimes you need them to be in a known location - not a known DNS address, but a known IP address. I cannot count the number of time I've had to "fix" a problem for a customer caused by PC changing address when it's hardcoded into something else - such as a door entry system or a copier/printer/scanner. Often these devices can only accept an IP(v4) address, and often the networks are not that well set up (I hate that) so DNS can't be used reliably anyway.
It seems that many people installing things like door entry systems and printers have this belief that PCs don't change address even when using DYNAMIC Host Configuration protocol. Among those that do realise what the D stands for, many seem to think that it's OK to get an address via DHCP and then statically configure that without doing anything on the DHCP server.
To be honest, even servers don't need a static IP
Really ? In a closed environment that may be the case, but once you consider the wider view then yes they most certainly DO need static IPs.
Take a web server for example. IF (and it's generally not the case) every site on it has the domain's DNS managed by the same people, then it would be possible to have the DNS updated every time it changes address - although the sites would go offline for a while between the IP change and DNS caches expiring). But also in the general case, DNS may be hosted elsewhere. And note that while www.example.co.uk could be a CNAME, example.co.uk (ie without the www) CANNOT be anything but an A record - I've had this discussion with supposedly professional website builders who've told their customers to have the DNS records both set to be CNAMEs.
Same applies to other services. For example, last year I had to change the IP address of a mail server at work - I was able to update it's canonical name in the DNS and almost all the customers didn't know anything had changed. But one was forwarding mail from another system which would only accept an IP address - and so their mail broke.
So a more accurate statement would be that in certain environments/setups, servers don't need static addresses.
-
Monday 19th March 2018 13:12 GMT Anonymous Coward
"It seems that many people installing things like door entry systems and printers have this belief that PCs don't change address even when using DYNAMIC Host Configuration protocol. Among those that do realise what the D stands for, many seem to think that it's OK to get an address via DHCP and then statically configure that without doing anything on the DHCP server."
Ah, the fun I've had, many moons ago, trouble shooting a printer issue, just to discover some idiot had it configured to use DHCP. Of course, back then, everything on then NT servers was static :)
-
-
-
-
Wednesday 14th March 2018 20:18 GMT Richard Plinston
Re: Oh dear
> Very poor practice to rely on static IPs
I was at a meeting with one of my client's network consultant. He was adamant that using DHCP (with reserved IP addresses) was 'best practice'. I disabused him of that idea - forcefully. The client had a workshop with several CNC lathes and flat-beds, each with a separate desktop computer where the designs were created and loaded to the lathe. I had provided each pair with a separate switch and fixed IPs. This catered for several failure modes in the network that using DHCP did not. It allowed the revenue earning to continue by catering for keeping the lathes and flat-beds running regardless. And, yes, later there was a power failure after which the servers did not restart.
The use of static IPs or not is not a 'one size fits all' situation.
> at least use DHCP giving out reserved IP addresses based on MAC address.
While that may solve some problems it leads to others. For example a failed network card cannot just be swapped and everything carries on as normal, it is no longer just a hardware problem.
-
Wednesday 14th March 2018 22:00 GMT Likkie
Re: Oh dear
"...Very poor practice to rely on static IPs, mostly by BOFHs who don't understand their network but it suddenly 'just worked' with static IPs. If you have to bodge, at least use DHCP giving out reserved IP addresses based on MAC address."
That sounds like some misremembered second hand advice that someone gave you about something else.
-
Thursday 15th March 2018 05:24 GMT donk1
Re: Oh dear
Really?
https://docs.microsoft.com/en-us/sql/sql-server/failover-clusters/windows/always-on-failover-cluster-instances-sql-server
"In a production environment, we recommend that you use static IP addresses in conjunction the virtual IP address of a Failover Cluster Instance. We recommend against using DHCP in a production environment. In the event of down time, if the DHCP IP lease expires, extra time is required to re-register the new DHCP IP address associated with the DNS name. "
-
-
-
Thursday 15th March 2018 10:44 GMT Amos1
Re: Oh dear
Especially when the corporate folks decide to run DHCP centrally via IP Helper statements on the WAN routers and a problem back at corporate (Backhoe work took out WAN backup line but who cares? That thing goes don all the time.)
And an overnight drunk took out the pole carrying the WAN primary line on the other side of the campus and all of the salaried people kept getting paid while the hourly people sat around waiting for their computers to start working in the plants.
And the best part? All of their after-hours alerting was done by email and Internet was run on those two circuits as well, including the connection to the DR data center. "Wow, we got like no alerts overnight!"
-
Thursday 15th March 2018 19:40 GMT Giles C
Re: Oh dear
Static up addresses are essential
Routers Bgp peering would be interesting... as would virtually any securely peered protocol
Firewalls, especially for nat statements
Access control lists - Cisco does not support dns entries in ACLs
Load balancers
Radius and tacacs servers
But not for
Client workstations
Wireless APs
To say you don’t need static addresses suggests someone who has never administered a network bigger than a single switch.
-
-
-
Thursday 15th March 2018 13:59 GMT Lee D
Re: Oh dear
Increase the DHCP lease times if it's a problem. There's no way a DHCP-providing server should be offline for more than a week, anyway.
Static IPs have their places, but generally speaking DHCP solves more issues than it causes. First, it usually means manually entering IP and subnet. As someone who inherited very odd subnets, I can assure you that it's not fun to spend ages on a problem only to discover it was a typo in an IP/subnet.
And DHCP with long lease times, failover DHCP server (bog-standard since Server 2008), and reservations for anything you care about makes life so much easier.
Honestly, how do you deploy clients? I press F12 in the BIOS, they get a DHCP address, it goes (via PXE) into WDS, I choose a Windows image, it reboots and when it comes up it's on the domain. Whether it's brand-new out-of-the-box, or a complete re-image. From that point on the address doesn't change unless the lease expires and for some reason it can't renew the old address (i.e. never). Literally one key-press, one mouse click and you're done.
How do you do that with static addressing without having to maintain huge lists of things and manually enter stuff in places, or code it specifically in templates of some kind? And when I need to find out the IP or force a particular option, image, etc. I can do it via DHCP management.
Static for servers, yes. DNS server you really have NO choice but to be static (it's silly for your DNS IP to constantly have to change, for instance, but you could easily give that machine a reservation on DHCP). Everything else will work no differently on DHCP or static and for every far-fetched scenario you can imagine on one, there's something on the other that works to your advantage.
But I honestly don't have the time to manage hundreds of individual client IPs like that. Set up the servers. Maintain a list of their IPs. Done. Everything else, you let manage itself because it doesn't matter. Group membership, web-filtering, whatever else you can conceive should then be set up on a computer-name basis, possibly a MAC address or authentication as a user, not by IP.
And in a world of VM's, BYOD, self-building clients and managed networks, its too much faffing to be bothering with any kind of individual IP address for services. I literally know two of my IP addresses off by heart - the gateway (.1 of my subnet) and a secondary DNS server (.10). Everything else I don't need to ever know and don't know and don't care, they're all named and DNS-accessible and worst-case I could find out via the switch (which also keeps track of DNS name vs IP vs Mac) in about two clicks.
I honestly haven't typed in an IP address other than those in ages (and that is to ping them to ensure they're back up after taking them down, so that everything else I'm about to do will resolve properly). I just reserve the IP on DHCP if I need to, e.g. I made an exception for web filtering on one machine, added it to an AD group, used that group for filtering I made a port forward to a particular machine, reserve the lease on DHCP and use the computer name. I can blacklist clients on RADIUS on the basis that their MACs aren't in the right groups, etc.
About the only other thing that needs static IPs is some forms of HA failover, but even that's not all of them and they're not "usual" IT stuff. DHCP, DNS, DFS, HTTP, SMTP, etc. failover doesn't need the same IPs on every machine, for instance. Just list multiple entries and have "server1.domain.com" or equivalent.
-
Friday 16th March 2018 08:01 GMT Maventi
Re: Oh dear
> But I honestly don't have the time to manage hundreds of individual client IPs like that. Set up the servers. Maintain a list of their IPs. Done.
Neither do I, or even have the time to maintain lists of IPs. It's called IPAM and most decent provisioning/lifecycle tools automate all that. Bootstrap new systems via DHCP (for PXE) then have them reconfigure themselves with the same static address as they build. Let the tool take care of assignments and managing leases. Easy as pie. If you need to bulk update changes such as DNS then that's what tools like Ansible are for; change one line in a playbook, test, commit, job done.
Servers shouldn't rely on DHCP - in fact servers should rely on as little external services as possible to sustain their operation.
-
-
-
Friday 16th March 2018 18:44 GMT SImon Hobson
Re: Oh dear
... at least use DHCP giving out reserved IP addresses based on MAC address
So every time one of your virtual server hosts gets rebooted, and the clients on it get a new IP address - since the default on some systems is to give clients a random MAC address. Oh what fun when the virtual server is a web server hosting hundreds of site, a significant number of which have their DNS hosted elsewhere.
As others have said, if you believe what you wrote then you clearly have had a very sheltered career !
-
-
Wednesday 14th March 2018 21:45 GMT Anonymous Coward
Re: Remember when Microsoft tested things before releasing them?
They've got form in inflicting these patches upon a bunch of volunteers, of which I happened to be a member of that group, without real testing. Three decades here now. You hear that they eat their own dog-food but that hasn't ever been true. They test on a few, select, configurations they use on campus rather than on a wider collection of systems that more accurately reflect "The Real World." That's left to the rest of us to sort out.
Hey, one thing that's nice about their new regime. Everyone else gets to join their eternal betas.
-
-
Wednesday 14th March 2018 23:08 GMT Anonymous Coward
"And as it is a new vNIC with a new MAC you'll get a new IP and have to go to the DHCP server to set it all up again... perfect!"
The vNIC should be the same unless the patch is able to reach out into the VMWare configuration to remove and then read the network adaptor. If it is just the NIC inside Windows that gets replaced (seems likely) then it will still have the same MAC address.
Please note though that I still don't agree with the original idiotic "DHCP is the answer to everything post".
-
Thursday 15th March 2018 00:56 GMT Anonymous Coward
Fix
Just had this while testing this months updates.
The fix is simple. Find the subkey with the old NIC configuration with the IP address you want to set under the following key:
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
Remove this subkey. You should then be able to set the IP address on the new NIC.
-
-
Thursday 15th March 2018 04:52 GMT Anonymous Coward
Re: Fix
Even before the drop-off in quality control of MS patches there seems to have been in the last couple of years we still deploy all changes in a test environment first, instead of blindly making changes to production machines.
This was why we found it in testing. We have now delayed these patches to production 2008R2 machines and will keep an eye on the KB for more information. We don't have that many 2008R2 servers left now though anyway, we have been migrating away from it. So far we haven't found any issues with this months patches for 2012/2016.
I was just providing a fix for anyone who found themselves in the situation where the patch had gone out to a 2008R2 server with a VMXNET3 NIC.
You have 300+ 2008R2 servers to manage? I hope you test changes to them first unless they are not doing anything important.
-
Thursday 15th March 2018 15:36 GMT Trixr
Re: Fix
Yes, of course you *hope* I test changes. I haven't even deployed these updates into our DEV environment yet. I always wait at least 3 days. Then there's at least 10 days before they go to Prod.
In my experience, we've been more affected and put more at risk by deploying crappy patches too fast than any runaway exploit.
-
-
Thursday 15th March 2018 10:56 GMT Naselus
Re: Fix
"It would help if MS didn't screw up *any* part of the TCP/IP stack or network configuration with a bloody routine patch."
Yeah, I've no idea why they keep doing this shit. There's really no reason for them to be poking around many of these systems, which have been working perfectly and to standard for twenty years, and which they're not showing any real sign of improving or changing for the better anyway.
For example, we found a couple of years ago that Win 10 breaks DNS subnet prioritization. Literally breaks it; it inverts the priority stack so that the last responding subnet is favoured rather than the first. Why? Why the hell would you need to poke around with that? It's been operating just fine since server 2003, and the mechanism by which it works hasn't changed and will never, ever need to change. The was no upgrade or change to the system planned or offered. But for some reason, some twat at MS was fiddling around in there and managed to screw it up.
-
-
-
Thursday 15th March 2018 05:37 GMT Anonymous Coward
Persist and we will retain the World as ours
Don't be Fooled, by the bullshit.
IVP6 is not supposed to allow class addresses A,B or C but, it does have the capacity to encapsulate IVP4 addresses, if everybody continues to use Class A,B or C IVP4 addresses encapsulated within IVP6 addresses then the servers will have to isolate those addresses just as in IVP4, and give us back the isolation we desire, that IVP6 stole.
Persist in this address usage and we will win. but give in like mindless Zombies and the world will be lost to big brother.
-
Thursday 15th March 2018 07:21 GMT arctic_haze
I stopped updating my Windows 7 boxes
In fact Microsoft did it for me demanding since January that I manually set a registry key to get further updates.
There are simply too many issues with all the
crapupdates of the last three months, including performance decrease and various side effects. Just see the description of the March Windows 7 bunch:https://support.microsoft.com/en-us/help/4088875/windows-7-update-kb4088875
-
Thursday 15th March 2018 10:17 GMT rmason
I just don't get it.
Why on earth do people go within a mile of MS patches until they've been in the wild for a few days and you've had a google/forum check etc?
I can only assume they're all new to the job. the mind boggles.
Use WSUS.
Calm down and see what happens first. Perhaps even consider some testing too.
entirely avoidable this stuff, in 90%+ of cases. it's probably 99%+. Not many situations necessitate same day or next day patching.
It's just lazy admins with auto update on.
-
Thursday 15th March 2018 11:00 GMT Anonymous Coward
Re: I just don't get it.
Depends upon the industry you are working in. For example the one i work in, requires critical security patches to be installed within 24 hours. We deploy to low impact servers and desktops when they come out, if all goes ok, move on to the rest. But if it fucks up, like it has been, for a few years now, since Microsoft 'changed' their testing regime, we have to go and fix the low impact servers and then decide on what to do with the installation on the remaining servers. If there is a work around or not.
Before the CU packs this was easier, don't deploy the faulty update. Now its all or nothing. We didn't deploy last months 2008 updates as it screwed up multiple things, and now as its a CU, cant update until we can find a fix /work around for the problem so now we missed out on this as the 2008 updates were not approved on the low impact servers because of last months update.
-
Thursday 15th March 2018 13:47 GMT Doctor Syntax
Re: I just don't get it.
"Calm down and see what happens first. Perhaps even consider some testing too."
Unless someone takes up your second piece of advice everyone will sit calmly for a few days, then, because there've been no reports of problems, everyone applies the patches on Friday - just in time for a week-end's panic.
-
-
Thursday 15th March 2018 11:07 GMT TrumpSlurp the Troll
Just a dumb query
On my W7 boxen I download the patches then wait a few days/weeks before applying them.
I assume Windows Update will ditch any failed patches held locally.
Then again I seem to recall still seeing the pre-release of the February bulk update in the Optional list when I had the real thing ready to apply.
-
Thursday 15th March 2018 11:25 GMT Anonymous Coward
Its a TRAP!!
This is just another cunning ploy to prod people away from using older versions of M$ software (that they can control), on to new versions that M$ control.
Just like all the crap thrown at Win7/8 users to force them on to Win10.
I have been hearing choppers all day, ever since dissing Trump on a US based forum.
(Trumplish - A baby talk version of US English - which in its self, is a simplified version of Traditional (British) English).
-
Thursday 15th March 2018 12:09 GMT psychonaut
smb v1
might be a cooincidence, but just had a call where an old nas box disconnected for no reason. turns out to be because it uses smb v1, and since yesterday, it hasnt worked. fix is to enable smb v1 in windows components (and also possibly check for ms client in network adaptor, turn off ipv6 and enable netbios over tcpip).
probably caused by recent win 10 patch
-
Thursday 15th March 2018 13:40 GMT JeffyPoooh
Once upon a time...
Decades ago, we had a large system with several HDDs. To give you an idea of how long ago this was, I believe that the HDDs were 5 MB (not GB, 5 MB). Anyway, it took me about 1.3 seconds to realize that we could install the boot code and OS onto *all* of the drives, and then set it to boot off the primary. If anything ever went wrong with the primary, then we could *instantly* switch it to boot from another HDD, and then gleefully carry-on with our day.
If I were running a server, then I'd install at least four HDDs (or four arrays). If anything went wrong with the primary drive (e.g. MS Updates), then I'd poke a button and boot it from the other HDD with the previous stable OS and then go for lunch. It'd be equivalent to 'Restoring' to the previous Restore Point, except being a hardware swap it would take mere minutes instead of rudely interfering with my coffee break.
There would be some details to sort out, such as ensuring that as much of the OS data as possible is stored on a different "Data" drive. So that the settings and such (Login) survive the switch to the alternate OS drive. One may have to do daily backups of the OS from one drive to the other. It might need like 6 or 8 HDDs to keep things organized, but they're cheap.
I can only imagine that MS Policies and OS design make this sort of thing very difficult these days. Too bad. Because the basic concept would make it nearly trivial to survive such disasters, while not missing lunch.
HDDs are fast, but not as fast as changing a C: to a X: and rebooting.
-
Thursday 15th March 2018 14:08 GMT Anonymous Coward
And there's a Word 2016 bug in the current set of patches
From: https://support.office.com/en-us/article/after-installing-kb-4011730-you-may-not-be-able-to-open-or-save-word-documents-bcbb5ed6-9246-4f3e-9572-f626dda01fc8?ui=en-US&rs=en-US&ad=US
After installing the March 13, 2018, update for Word 2016 (KB4011730), you may not be able to open or save Word documents.
This issue occurs only for those who receive Office 2016 updates using Windows Installer technology (MSI). If you have a Click-to-Run edition of Office, such as Office 365 Personal, you won't encounter this issue.
STATUS: WORKAROUND
We are aware of this issue and working on a fix. In the meantime, you may be able to workaround this issue by installing the March 6, 2018, update for Office 2016 (KB4018295).
-
-
Thursday 15th March 2018 19:19 GMT Lion
It is like spitting into the wind
I thought those optional monthly previews are intended for those with a test environment to do a trial run on the upcoming monthly cumulative patches before the important and checked one gets sent out by windows update. Is it not being utilized for that?
Non-enterprise users are very unlikely to have test systems, so sending monthly previews to them is a temptation that they could do without.
Windows update set to automatically install is like spitting into the wind.
Windows Update is integral to WaaS.; this model is the future of Windows.
- The corporate vision, based on spitting into the wind.
-
Thursday 15th March 2018 22:51 GMT Anonymous Coward
Quality testing
Wife has updated her lappy. Now gets 0xC000001
Be great if M$ had to pay $10 compo for every machine they borked. Watch that share price drop and QA costs increase.
Having said that, from a professional point of view, M$ have helped me pay my mortgage over the past 20+ year, put kids through uni, PHD plenty of £££’s in the bank.
Hmm, in retrospect, as you were....
-
-
Monday 19th March 2018 17:19 GMT Richard Plinston
> some real "developer" tossers out there who STILL lock their license keys to IP address.
I wondered how that could possibly work. A home computer on a dial up modem or ADSL, for example, could get a completely different IP address from their ISP every time they connect. With a network it is trivial to change IP address and several machines could have the same address as long as they don't try to communicate.
The answer seems to be that they _don't_ lock the licence to the computer's IP, it is the licence _server's_ IP that is locked to the licence. The server can control how many machines are using the software.
"""The license key delivered to you must be converted to a permanent key that is locked to the IP address of the computer that runs the DialOut/EZ License Manager."""
So the license server should have a fixed IP, or even a reserved IP on the DHCP server, but the clients running to software may not need to have the same IP each time.
I did have some software that was licensed to the network card MAC address. Did they not know about ifconfig ?
-
-
Wednesday 21st March 2018 08:37 GMT kiwistylez
Hi Everyone
Please try the following and let me know
Remove this reg key
Get-ChildItem "HKLM://System/CurrentControlSet/Enum/PCI/*/*/Device Parameters/SlotPersistentInfo"|Remove-Item
Then install either kb4088875 or 4088878 and reboot.
I have tested on 6 machines and nic config are still there once server has been rebooted
Info Article about cause of Nic issue
https://kb.vmware.com/s/article/1020078
Let me know how you all get on.