Management software for IBM's DS3000, DS4000 and DS5000 series storage arrays is waking sysadmins at 2:00 AM, thanks to an obscure software setting. As detailed in the Aussie Storage Blog, some versions of the Profile Collector feature in Big Blue's DS Storage Manager package schedules data collections at 2:00AM. The blog's …
Checks versions at office.
Sets every PC in enterprise to wake at 13:50AM.
Slinks off, grinning.
".....wake at 13:50AM."
What planet is your time zone on?
hahahahaha damnit, yes,silly of me, I didn't even notice until you pointed it out.
Same issue with the XIV a while back...however the XIV node would go offline....which would then pass the workload onto the next node and then the next. Good game of dominoes!
The problem here is software which assumes it knows best about what it installs. This ranges from putting an icon on your desktop you don't want to installing crapware or spyware which you definitely didn't want. Some software hard-codes application paths, but the worst culprit is itunes which, without your knowledge, installs a bunch of services, including one which harvests the media on your hard drive, bringing your PC to a standstill. This is the #1 reason itunes is banned on work machines.
In this case, (unlike itunes) you can deselect the offending software but it should really ask if you want to install rather than the other way round. The worst thing is the software doesn't even need an installer. A zip file would do.
Storage Manager is bloated anyway. It's mostly a JRE.
As for collecting data, you can always use a script with a scheduled task in Windows. I'd post how to do it it here but I doubt anyone really cares.
I've seen this before with other software
Any software which triggers a load of work at the same time on multiple systems can cause this type of problem.
And don't pick times like 1:00, 2:00, 3:00 because some other software is probably using those times (Windows update with a default of 3:00, default times used by backup products etc). Stagger them instead.
And the experienced sysadmin might collect alerts and deliver them via SMS at say 6:00 am.
Re: I've seen this before with other software
If one of my admins collected alerts to be delivered at 6am, he'd be looking for a job working for someone else.
If, on the other hand, he recognized that the client application developers were (and probably still are) idiots, and reconfigured things accordingly, he'd find himself less often on-call.
Re: I've seen this before with other software
"If one of my admins collected alerts to be delivered at 6am, he'd be looking for a job working for someone else."
Not so fast.
It depends on your SLAs and what cover you have with your hardware support folks. When I was an admin for a large development and testing environment we only had hardware cover from 7am onwards. A 6am alert was quite adequate to get in early and either fix things yourself or place a hardware support call.
Production systems were a different matter of course.
Being a storage admin of IBM DS4800 boxes I'm usually already awake at 2.00am worrying about the integrity and availability of the data on those buckets of dog poo. (or actually restoring the data , or installing this weeks firmware, or just crying.)
I wonder if NetApp knew what they where in for when they bought Engenio . . .
It isn't ONLY that IBM gets this...
At our site, we have a renowned SAN vendor that denied this syndrome. We ran a benchmark that showed DRAMATIC dataflow reductions at the two start-times of our enterprise backup system. We saw I/O go from x to 6x in the early evening AND from x to 9x at 2AM (when the big systems did enterprise backup).
That vendor denied it, even after we showed the benchmark output. The benchmark was absolutely reproducible, too. That is what caused us to bring in OTHER vendors for SAN equipment. At least, those vendors were up-front about this kind of problem.
By 'x' and '6x' or '9x', I mean...
In my previous post, 'x' was the amount of time to do the 15GB data transfer (disk-to-disk). That same data transfer went up by a factor of '6' and '9' when the Enterprise backup system schlepped the data from local-dump files off to the Enterprise Backup system...
- HALF A BILLION TERRORISTS: WhatsApp encrypts ALL its worldwide jabber
- HUMAN DNA 'will be FOUND ON MOON' – rocking boffin Brian Cox
- Bang! You're dead. Who gets your email, iTunes and Facebook?
- YOU are the threat: True confessions of real-life sysadmins
- Blackpool hotel 'fines' couple £100 for crap TripAdvisor review