back to article A ghoulish tale of pigs, devs and docs revived from the dead

"My pages have come alive!" accused one from my pod of guinea pigs, unfeasibly. This (as it turned out) not-so-singular anecdote from my murky professional experience working on large-scale content management projects came to mind this week while reading The Reg. It was triggered by that recent Who, me? story about the little …

Silver badge
Joke

So...

The whole experience was 'Just like you imagined' it would be ?

7
0
Reply

"every smooth operation is a cupboardful of IT bods frantically conducting a real-world and real-time disaster limitation exercise"

Shss, don't tell everyone, it our little secret.

37
0
Reply
Silver badge

Secret is out

@Bob, one of your mates must have spilled the beans. I asked an IT manager how things are going...

IT: "Like a duck"

ME: "What the fsck?"

IT: "You see us gliding gracefully across the lake. Underneath the feet are paddling like hell"

36
0
Reply
Silver badge
Headmaster

Re: Secret is out

That's swans.

Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.

Although I've known projects where the latter would actually be more of an accurate description...

39
0
Reply
Silver badge

Re: Secret is out

Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.

Err.. not mallards. They don't dive at all (what they do is called "dabbling" - which means their head is underwater but their tails are stuck up out of the water, pointing at the sky). Some ducks (crested ducks for example) do dive but not for very long.

7
1
Reply
Happy

Re: Secret is out

Nobody told this one - https://www.youtube.com/watch?v=M-KqMsUpekU

2
0
Reply
Silver badge

Love it!

You've confirmed some of my deepest, darkest suspicions about what really happens when I grovel and beg for old data to come back.

Sometimes, though, it comes back on its own. Like my night of the living dead:

Supposedly in this one govt agency emails - once purged off the exchange server- are gone forever. Of course that's utter BS because if management wants to pop you for misconduct they can suddenly get not only what you've written, but even stuff you only dreamed of after a few stiff drinks. Mere mortals do not get their lost emails restored.

Imagine my surprise when I opened up outlook after a prolonged server outage and magically found my mail spool transported ten years into the past. Instead of my working set of spam, I was seeing one specific day in 2005 all over again.

Here's the part that keeps me awake: email on the top was in all caps, title screaming: "YOU ARE LATE FOR A MEETING!" Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that.

So where the hell is he, and am I late for a meeting upstairs or down?? Is the weather uncomfortably warm??? Speak, ghost of the server, speak!

56
0
Reply

Re: Love it!

Dear Chairman of the Bored

I have to inform you that not only the 'pages are alive', that 'emails get resurrected' but also there are ghosts in the machines - how else do you explain these damn systems working one moment, then failing then working again without anyone even touching them. I swear the software is taunting us.

In addition to that, it has been known for many years in the highest level of BOFH'dom that the servers also have gods that not only require, but demand blood sacrifice from any mortal working on them in the machine halls.

yours faithfully

IT bod in chief

19
0
Reply
Anonymous Coward

Re: Love it!

"[...] but also there are ghosts in the machines[...]"

A large interactive system in the 1970s. The main administration console suddenly types out "I am the ghost of the machine - I have changed your password". Unfortunately the prankster had made a programming error and the message repeated every few seconds - preventing the admin team from logging in.

He was fired the next day.

19
0
Reply
Silver badge

Re: Love it!

That's dark and scary. Think we need to start drinking immediately.

11
0
Reply

Re: blood sacrifice

We have found a certain degree of success in utilising the blood of users. You may just require more of it thereby creating a win/win situation. The appropriate oath also appears to still work, you know the one: "Why won't you WORK you &*%!ing piece of £$(^!!!" at around 120 dB.

14
0
Reply
Silver badge
Terminator

Re: Love it!

@ Chairman of the Bored "Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that."

Was the death, cough, an accident? Did you go all BOFH on him by the top floor windows that have the broken catch as retribution for him typing all in caps?

15
0
Reply
Silver badge
Coat

Re: Love it!

@Bob Wheeler

there are ghosts in the machines

Nah, it's the Phantom of the Operating System.

23
0
Reply
Silver badge

Re: Love it!

ghosts in the machines - how else do you explain these damn systems working one moment

Mostly, for me, it's the other way round. Stuff won't be working at all until I go and stand next to the computer.

At which point, it'll work perfectly.

Maybe it's my sparkling personality (bought on Ebay) or maybe They Know that in my pocket is an etherkiller and I'm not afraid to use it..

10
0
Reply
Silver badge

Re: blood sacrifice

We have found a certain degree of success in utilising the blood of users

Sadly it seems to require the blood of virgins and they are a diminishing resource round here..

7
0
Reply
Mushroom

Re: blood sacrifice

@CrazyOldCatguy

"require the blood of virgins and they are a diminishing resource round here.."

Nah, all of our PFY's are virgins, ... oh wait, you meant the female kind....

15
0
Reply
Silver badge
Facepalm

YOU ARE LATE FOR A MEETING

Because email is famous for getting someone's attention in real-time

13
0
Reply
Silver badge

Re: Love it!

"demand blood sacrifice from any mortal working on them in the machine halls."#

That's why the metal-work has sharp edges.

11
0
Reply
Silver badge

Re: YOU ARE LATE FOR A MEETING

The best reply is "Yes I know. I'm busy working through all your emails about meetings".

9
0
Reply
Silver badge

Re: Love it!

Working in the trenches in IT is like living in real time two movies at once: "Groundhog Day" and "Ghost Busters".

10
0
Reply
Silver badge

Re: Love it!

Trust me, the thought crossed my mind. Actually I was more focused on how to make the body disappear. Question - if you shoot a guy for ranting in all caps ... Do you need to use a silencer?

12
0
Reply
Silver badge

baked bean wrestling

Watching the Ann Margret scene out of Tommy before bed again, Dabsy?

8
0
Reply
Anonymous Coward

Scarily true to life...

I read this as I am trying desparately to build a USB drive with the latest UEFI version for my three Lenovo System X VMWare hosts to unfuck the fuckup that was Lenovo's first iteration of Meltdown/Spectre back in January. Just trying to get the latest UEFI from their site to a bootable device is an effort of ridiculous proportions - the default option gives you an older version of it rather than the current one.

Meanwhile, all my VMs are running on two hosts, and whilst not quite hitting resource limits, I've not got the comfortable feeling I'd like to have...

14
0
Reply
Silver badge

Re: Scarily true to life...

And that is the problem with virtualization - you can wipe out more than one server in just one go... and data recovery is well nigh impossible.

Backup, backup and backup.

Just not to Tier 4...

https://www.theregister.co.uk/2014/11/01/bofh_2014_episode_10/

7
0
Reply
Silver badge

Re: Scarily true to life...

desparately to build a USB drive with the latest UEFI version

I spent some considerable time recently trying to install Windows 7 on an old 17" MAcbook Pro I had lying around (my wife wanted a laptop to do Excel-type stuff on but simultaneously didn't want to spend actual money on one..).

Even wiping *everything* on both internal drives and booting from a clean USB or CD install didn't do it. Which is bizarre because (on that self-same Macbook Pro) I used to dual-boot into Windows 7 quite happily.

So, either there's been a EFI update applied to the MBP that borks the boot process or the latest incarnation of Windows 7 won't work at all on a 2011 MBP.

Or, knowing my luck, both. Need to find my *old* W7 boot CD. Trouble is, it was done on a cheap CD-R so probably won't be readable either.

4
0
Reply
Boffin

Lesson

Surely there is a lesson to be learned, and it is the remarkably simple one of "Test everything thoroughly before you go live, and don't go live until you know it works"?

Call me naive if you will...

GJC

15
0
Reply
Silver badge
Mushroom

Re: Lesson

Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time. This product will lack pretty much all the features that differentiated it from the previous version during the procurement process and if you are very lucky will at least do 80% of what you need it to do.

Then it breaks and the budget put aside for enhancements rapidly becomes subsumed into fixes and maintenance.

Then the cycle begins again - lessons may be learnt but as only 2 people were around from the last time and are still suffering PTSD it is unlikely to break the software cycle.....

15
0
Reply
Anonymous Coward

Re: Lesson

"Test everything thoroughly before you go live, and don't go live until you know it works"?

As Donald Rumsfeld said - there are always "unknown unknowns". In other words - expect the unexpected.

The live environment is by definition different from any test environment - no matter how accurately you believe it to have been staged in a dress rehearsal.

What has often happened in the past is that on the "live" day users start reporting problems as requested. What they fail to tell you is that the problems existed previously but no one had fixed them.

I remember two major switch-overs.

1) A large online system was being moved to the next generation hardware - and running a different O/S. This involved virtual machines and all sorts of emulations so that no customer programs needed recompiling. Several of the user sites had resisted the advanced testing of their connections - but had eventually agreed. On the "live" day the reported problems were all previously known to the users on the old system.

The plan had been to keep the old system as a warm stand-by for a couple of weeks. After three days the customer asked for it to be switched off.

2) a customer had three mainframes - each with their own comms lines. The project was to use one front end comms processor to allow any terminal to switch between mainframes. The weekend before switchover each comms cable and its port were marked with numbered sticky labels. Different shapes and colours differentiated the three mainframes. The cables were then plugged into the new ports.

On the day we were ready to move all the cables back if necessary. Only one real fault was reported - and that was for a new comms port that the end user site had refused to allow to be pre-tested. The senior engineer said "I would rap the knuckles of anyone doing this" - as he took the port board out of the live rack and put in a spare. It worked. The removed board had a repair label saying "NFF" (No Fault Found).

Over the next few days quite a few problems were reported - and they were all pre-existing conditions that the users had previously endured on the old system.

6
0
Reply
Mushroom

Re: Lesson

Unfortunately out here somewhere, we sometimes find ourselves in the situation where the old license runs out before the replacement software is quite ready for primetime - but we have to go live anyway...

5
0
Reply
Bronze badge

Re: Lesson

Ha. Haha. Hahahahahhaaa...

I was talking to a friend recently, who's working for a company which is about to switch some Very Important financial infrastructure into the cloud.

However, the people doing this work have decided to not tell anyone about this, to prevent the usual "the sky is falling" claims which tend to abound when any changes are made. After all, they've cheerfully proclaimed, everything should Just Work after the changeover.

Oddly though, the list of people who haven't been engaged in the handover process includes the internal QA team. Nor is there any evidence of a contingency or rollback plans.

Overall, it does seem like an appropriate time to order some popcorn and get comfy on the sofa...

15
0
Reply
Silver badge

Re: Lesson

Call me naive if you will...

Hi naive.

I presume that you've never worked in a large IT department. That sort of thing is depressingly familiar.

A combination of restricted resources, senior management unrealistic[1] deadlines and general "if anything can go wrong it will"ness will give you it every time.

[1] Surely an oxymoron..

4
0
Reply

Re: Lesson

Why did the fist lines in Your comment made me think of C-beams?..

3
0
Reply
Silver badge

Re: Lesson

"Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time."

This is a lesson that needs to be learned by manglement. It never happens because when you know you're always right, why would you have to learn anything.

5
0
Reply
Silver badge

Re: Lesson

"they were all pre-existing conditions that the users had previously endured on the old system."

They probably realised they'd a chance of getting them fixed if they reported them afresh.

7
0
Reply
Silver badge

Re: Lesson

Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither.

Usually what happens is "neither". Seems manglement only see the "go live day" as the goal. They'll hammer with "improvements" and "features" to the point that the "testing" schedule gets tossed out the window as no time is left. In an ideal world, if the testing start date slips, so should the "go live date".

6
0
Reply
Bronze badge

Re: Lesson

Which is why we are stuck with windows 10 instead of a proper rev of windows 7.

*hides in the anti-nuc bunker*

3
0
Reply
Bronze badge
Headmaster

Re: Lesson

"Senior management unrealistic deadlines" isn't an oxymoron, it's a tautology.

3
0
Reply
Boffin

Re: Lesson

For the record, I've been working in and with large IT projects for over three decades now.

I understand that testing is frequently done supremely badly or not at all. Which is why the lesson I pointed out should be learned by anyone who have been involved in such a clusterfuck.

Yeah, I'm an idealist. So sue me.

GJC

2
0
Reply
Silver badge

Re: Lesson

it is the remarkably simple one of "Test everything thoroughly before you go live, and don't go live until you know it works"?

"Always mount a scratch monkey" is simpler yet.

(N.B. Anyone interested in the corrected version of the scratch monkey story should refer to the first link in the Jargon File entry linked above. Or that link in the previous sentence.)

2
0
Reply
Silver badge

sympathies

"i.e. near the chilly windows that didn't shut properly."

Which is also the same one that the sun mercilessly beams through in warmer days.

Ah, to live with my friend 'the big shredder' and all the other 'corner' occupants again . . .

8
0
Reply
Bronze badge

Back in the 90s...

There was a demo for some of this newfangled internet technology.

However, as the technology was still a work in progress, the demo involved a certain amount of smoke and mirrors. And the person doing the demo had been given clear instructions about where the smoke and mirrors were placed.

Alas, said person was nervous and strayed from the script. Cue some furious hackery in the background as the tekkies supporting the demo had to try and fake up the S&M elements in realtime.

The main tekkie in question is not a small chap; he'd have made a fine rugby forward or viking berserker. As such, when the demo-presenter cheerfully came over afterwards and proclaimed "well, that all went well, didn't it!", he very nearly got throttled...

10
0
Reply
Anonymous Coward

Re: Back in the 90s...

Same thing in the early 1970s. A roll of pre-punched papertape in a teletype in the mainframe data centre. Ready to be switched back-to-back with the one in the remote press/bigwig hotel venue. A very experienced operator at the demo end - who knew not to deviate from her script.

For the first time ever the system ran without fault - until a few minutes after the demo ended. It was several weeks before such reliability became standard.

7
0
Reply
Silver badge
WTF?

Re: Back in the 90s...

"S&M elements"

6
0
Reply
Silver badge

Re: Back in the 90s...

fake up the S&M elements in realtime

I hear there's quite a market for that in some parts of Soho..

3
0
Reply
Facepalm

This is what happens when...

..."Management" publicly promises a live date in six months, before any problem analysis has been done. Or even asking the IT department how feasible it is.

Having worked for a major UK Telco in their development department, I got really sick of that shit; it precludes anyone doing a decent job, and leaves you with "the only thing we have time for is the shittyest quick fix possible".

5
0
Reply
Anonymous Coward

Re: This is what happens when...

I have a bit of code here at work with a comment before it that says (and I quote!), "A seriously egregious hack, but we have to do it..." Of course, I'm the one who wrote it... and it really had to be an egregious hack in order to function properly.

(AC 'cos actual comment in actual code from $employer)

Of course, my favorite is "TEMPORARY, I HOPE HOPE HOPE" in the final version of the Apollo 11 sourcecode. Apparently it worked.

15
0
Reply
Silver badge
Happy

Re: This is what happens when...

Here it is: LUNAR_LANDING_GUIDANCE_EQUATIONS.agc (link is to github.com). See lines 179 and 180.

9
0
Reply
Silver badge

Plan F

Plan F is the one in which you screengrab thumbnails of lost documents from the CMS browser window

Nice. Our plan F is to kill all witnesses, throw the bodies to pigs (after removing teeth) and pretend nothing really happened in our space-time continuum..

15
0
Reply
Angel

Re: Plan F

@Potemkine

Some plans need constant attention and tweeking.

Your Plan F is faultless and is the ultimate expression of BOFH'dom

8
0
Reply
Silver badge

Re: Plan F

Unless he also has a solid plan to dispose of the teeth (and the blood pulling teeth, even from a corpse, no doubt involves) it's not all that solid.

4
0
Reply

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018