I've had three buttons to press in the last 500 light years...
and that was just to put the coffee machine onto manual. Or onto THE manual, if you prefer. ;)
"It's just a small change!" the Boss whines. "It's not a small change, it's a Friday afternoon change," I say. "We don't do those. We do Monday, Tuesday, Wednesday and Thursday afternoon changes if you want, but not Friday. If it's urgent we might do a Friday morning change, but it has to be pretty urgent. Lives must be at …
Friday afternoon changes are the best. Just make sure that:
-you strictly the required change, for which you have a written order, 2 minutes before the end of the shift
-you're not on call on the week-end, or in on the following Monday, or on Tuesday for that matter. On company-approved leave, of course (to be made up during the week-end, if need be).
-the change has the potential to ruin someone important's life (or contract) if anything goes wrong.
You only usually get the one try, so make it good.
Friday afternoon requests should be pretty seldom after that.
I used to work for an electronics company that had QS 9000 certification (for use in cars) and one of the rules was once you started testing something you had to keep testing it, there was nothing about stopping testing. Which was really stupid when you realise one of the things being tested was the ambient temperature in a tin roofed shed used for storing plastic moulding compound. Yes the compound needed to be kept within a certain temperature range, no there was no chance in hell of ever maintaining that temperature range during summer with the tin roof in direct sunlight.
You actually followed the procedures?
When I worked for a number of ISO9001 certified companies in the 90s we told the auditor we followed the procedures and then did whatever was required to get the job done.
In one job I inherited the procedure manual from a previous employee and it was all complete bull. Nothing I did was actually in the manual and nothing in the manual was actually part of my job. The manager was non-technical and didn't have a clue. About 2 weeks after I started we had an auditor visit so I told my manager the manual was up to date and arranged to be off-site visiting clients all day.
"You actually followed the procedures?"
"When I worked for a number of ISO9001 certified companies in the 90s we told the auditor we followed the procedures and then did whatever was required to get the job done."
Ah, see, it's probably not so bad if auditors are the only ones you have to worry about. Back in the mid-late 1990s I worked for the sort of company that used ISO-9000 like a weapon against vendors, to force them to do everything to the letter of our agreements.
In our defense, we were spending MILLIONS per... week, probably, at the height... contracting 3Com/USR to build 7' cabinets full of their hardware, outfitted to our exact specifications, that we could drop into data centers and machine rooms all over the US, and eventually much of the northern hemishpere, as part of building AOL's network.
(Yep, that was how it was done back in the time of landline dialup and long-distance toll charges — providing local-call access numbers in every calling region of the country meant taking the modems to the users. So we installed anywhere we could get space. Ideally in local data centers, but in areas where that wasn't an option, anywhere we could get space and have circuits delivered. Hotel basements, office park machine rooms, newspaper offices, you name it.)
Because we placed the hardware exclusively in offsite locations, and hands-and-eyes maintenance and troubleshooting was often done by local techs (or non-techs) who knew nothing of our equipment, we had to be confident that we could describe the exact location of every card, port, cable, and screw in the cabinet, and that every cabinet in the field would be exactly the same.
So, our head of procurement (a perfectly lovely woman, but also a holy terror whose bad side you never wanted to find yourself on) justifiably freaked out whenever the builders got ANYTHING at all out-of-spec. Even, or especially, the little things that "didn't matter", like using the wrong color or length RJ45 cable for an interconnect. (We had cable kits custom-built for the cabinets: Some 40- or 50-odd cables in seven? distinct jacket colors, each one's length specced down to the inch so it ran the exact distance from point A to B, routed properly, with no excess.)
She'd immediately start speed-dialing her way up the chain of command over at 3Com, or in the worst cases head straight for the airport and get on the next flight to Chicago, brandishing SLAs, ISO9000 certifications, and purchase contracts like a sword, (metaphorically) beating their build teams over the head with them until the build quality was pulled back into alignment with our specs.
It could be kind of awesome to witness, honestly, just as long as you were standing behind her instead of in her crosshairs.
I don't understand. Surely this is the best time to make changes that go horribly wrong. At least you've got two days of oblivion drinking first, before you have to deal with it. And if you're lucky, it'll all sort itself out, someone (not you obviously) on call-out will have to deal with it, or nuclear armaggedon / asteroid strike will happen first.
No, the ONLY time you make a small change on a Friday afternoon is when that Friday is the last day you will ever work for that company. Any other Friday, the mess you make on a Friday is the mess you'll get to clean up the following Monday, only it'll have had two whole days to fester and ferment and get nicely vile and ever so much more difficult to clean up.
No System Changes On A Friday.
This is usually followed by spending Friday night and most of Saturday undoing that 'small change' and the subsequent systems/processes that were affected/ground to a halt.
The requester of that change had a wonderful weekend however after he was told it didn't work as planned and merely replied "Oh, OK, just undo it and I'll have a look at it on Monday. Cheers."
Several pints to those of you who have the authority to push back on these last minute requests.
Thats how it starts... and then comes the "well that is odd"
Several years ago, we needed to replace the battery for the RAID cache in one of our ProLiant servers. No big deal. Done it many times and I said I would do it on a Saturday evening.
My wife and kids had come with me as we were going to go out to eat after the 30 minute job.
The problem was, this server had another server resting on on top of it with wooden slides from an old wooden desk as spacers for ventilation.
Since I was going to have to shutdown the one server to move it, I was going to use this opportunity to install rails (not the exact ones, but they will do the job) so we wouldn't have to do this again.
Severs down. Battery replaced. Rails mounted and server put back in rack.
The server with the new rails doesn't respond to the power button.
HP says the service contract has expired and there is nothing they can do.
Ended up having to meet one of the Wintel guys from the data center (that is about 3 hours away in another state) in the middle of the night as he brought a replacement server.
I told them never again and ever time they want me to touch a server I ask them to verify the maintenance contract.
My family had to settle for cheap take out that night...
1) It means that if there is an undetected problem then it will show itself before the weekend so worst case scenario we can spend all weekend fixing it without causing significant damage to the company processes.
2) It means nobody notices when we get them all done before lunch and bugger off.
It is sold to the relevant authorities as ensuring that we do our jobs efficiently and making sure we're the only ones that suffer if we get it wrong. It's sold to our department as early finishes most Fridays. Both sides feel smug about the deal and I can't remember the last time it actually inconvenienced us.
I guess your business is not 24 hour....
We usually find changes on a Thursday prior to a bank holiday weekend only start to show as problematic at 5pm on Friday, when all the people who know the details and how to roll them back are generally escaping and uncontactable.
We are getting new procedures soon....
"we can spend all weekend fixing it"
And there's the kicker. And the reason all of our changes are done prior to Friday. As soon as my (and almost everyone else's) clock hits 6pm, the office ceases to exist, and doesn't reappear again until 09:30 Monday morning.
Just one of the reasons I'm a coder now and no longer a PFY :-)
"we can spend all weekend fixing it without causing significant damage to the company processes"
You've been sucked in there.
Changes are made when people are around to notice them and people (you) are getting paid to fix them, why the hell would you spend your free time doing the corporates a favour? You can be damned sure they won't do you any.
Reminds me of some fuckers at a place (a large, global bank) I used to work in a former life. Staff used to be paid additionally and well for work during their on-call duty. Some of those would deliberately implement some small fuck-up, e.g. in a batch run, that would go off during their on-call. They would have safed the
IT bank once again, they would be paid well, and they probably hated their partners.
Once found out the incentives were changed to something more normal: paid for on-call duty but not for actual work during this time.
Since working in corporate land I've heard the term "Reach out" alternated with "Touch base with" more times than I care to remember, meaning send an email and forget about it.
I joined their club and started using the terms, but now my meetings end up sounding like a Depeche Mode song as I "Reach out and touch base..."
Ah, I remember this from a surprise tax inspection.
Standard procedure is firstly to panic.
Second to get a dedicated laser printer and some coffee (some to drink, and some to put convincing stains on the new documents in order to avoid them looking quite so fresh and new. That, a bit of crumpling and a few staples give a nice convincing air of versimilitude to your new creation.
Then station the nice tax inspector on the cushiest office available, on the top floor, with tea and biccies. Hoping that they'll ring down for the documents they want to see, rather than descend and climb 4 flights of stairs, giving you sufficient time to print what's missing.
There was no fraud on my part here. Just that the company's backup strategy for the accounts had been to, erm, not do any backups of that particular server. Ooops. I was one of the people brought in to fix it, and I'd reconstructed it in Excel, just not in the accounts software or the files yet.
As stated above. No change Fridays.
Devs are gone, management is gone, and you are expected to get it all done and resolve any issues.
No thanks. Did that for years at a shop. Devs didn't test their code well enough often enough that it was just predictable that things were going to sideways once we deployed. And it was just as predictable that they would be impossible to get ahold of for a good hour or more. On a Friday or Saturday. Then you have to wait for a fix. Re-deploy. Oh, and all of this on site. Just in case...
Now, if shit breaks over a weekend for some other reason, well, I'm there and will ride it to conclusion. But planning on ruining a weekend? Fuck off. :)
A task is never simple when it's delegated. You'll be told it is, but if it was, it would already be done.
Many things in life are simple. They are also not easy. Win the game by getting more points.
I'm happy fixing messes, and crazy stuff. Just don't sell me on it as simple :)
The only changes I'd accept on a Friday is a live test of new premises. Before Monday first day in new digs, you stick 10% of the bodies in Friday at the new place, so you've got the weekend to fix stuff so people don't lose their biscuits.
The issue with any other changes is getting hold of any random person is exponentially harder than on midweek nights. So when it goes bad it goes worse, and your weekend with it. Or someone will miscongfigure your settings and turn their phone off so you learn to not do stupid fucking things like request changes for Fridays.
I never minded working weekends. 4 days for 50 hours, and have a "weekend" of Monday to Wednesday. Plus all the booze you can drink. Being a chef is great, apart from the RSI.
Simple changes and simple jobs for the most part only exist in the minds of simple managers.
People who fail to really manage because they have no concept of what may be involved in carrying out a 'simple' request, to them it is simple because all they need is sufficient language skill to articulate the request before they go back into stasis.
Our code librarian/help-desk coordinator released a fix late on a Friday afternoon once, before hopping a plane on an out-of-the-country trip. Unfortunately, I was the PFY appointed to take over while he was away for the next two weeks. At about the time his plane began the takeoff roll down the runway, the installer installed the fix, and promptly shutdown a major code development site (1000+ developers!). And, of course, they all immediately started calling me to fix the problem. AIEEEEE! Talk about being in the hot-seat.
"Please sign here for the out of compliance hours change".
PFY - disables alarms and pulls mains on server so its running on APC while you distract bosses attention.
tell the boss you have made the change 5 minutes before the end of the day when you know the APC will run out of charge 6 minutes later.
retire to the PUB.
wait for the calls and pleading from the manager for you to fix the problem.
Monday morning reboot the server and tell the BOSS it took all weekend to fix. present him with the overtime bill and when the director asks what happened and why he has such a large bill for a small change present him with the Bosses signed change request.
one new boss :-D
I know I have some former colleagues on here, so who remembers the day the US IT department did some changes just before midnight their time and went home? Leaving those of us just starting work at 8am in the UK without a functional network... I remember hearing the boss telling someone "I don't care what time it is there, get them back in to fix it".
There was a theory that cars built on a Friday (and a Monday) were likely to suffer quality problems (probably debunked by snopes), but the phrase Friday Car was popular. The Swedes have the word måndagsexemplar, which translates as Monday items for the same thing.
Humans should only work a 3 day week just to be on the safe side. Actually, come to think of it I think most of them do.
Our policy states they must log a HD no later than 48 hours before the person starts, so they logged them at 4:50PM on Fridays and we had to rush around creating AD accounts and all sorts of fun on Monday morning.
Until we got our current CFO and he blew up in front of the CEO about it.
Biting the hand that feeds IT © 1998–2019