Ah, but a friend of mine actually knew Chewy. I got the story from the horse's mouth. As it were.
Not a splitter, a reformist.
Welcome to the fifth edition of Who, Me? It's a new Register weekly column in which readers confess to times their skills fell just a little – or a lot – short of what was required to stop things going pear-shaped. This week, meet "Dom", who back in the 1990s learned the "Clarion" programming language. Which wasn't much use …
Yonks ago, in a different life, I and a colleague got the task of installing some newfangled twisted-pair to replace the coaxial cabling. The offices had nicely accessible, large cable ducts running along the outer walls. And all the outlets, for power, network, telephone, were on the bottom of the duct, with the connectors being plugged in from below - and against gravity. The only obvious problem with those duts were their lids, outright bitches that heavily resisted both, being taken off and put back on. To put them back an aweful lot of fist-hammering was used. Acquired my "iron fist" there in Mr Miyagi-style.
So, one day while happily hammering along, suddenly the face of an accountant went pale, then white. Just when I was starting to worry that he might be dying he turned redish - steaming hot red! Accompanied by a screamed "I lost all of my day's work!" - it was mid afternoon.
That was about when we realised what happened: the hammering slowly but surely drove the power plug of his computer out of the mains socket. An orange socket, the ones connected to the UPS. One of us, I can't remember if it was my colleague or me, calmly said to the destroyed, steaming accountant: "you should regularly save your work - you never know when there's a crash." We left the office, closed the door and in the staircase we got the mother of all laughing fits.
> Accompanied by a screamed "I lost all of my day's work!"
I was told the story of an electrician who popped a circuit breaker in a university department to change some wiring in a few offices only to hear some screaming from along the hallway and see an academic run into the hall screaming "my work, my work!". It turned out that he was working off floppy and hadn't saved back to disk for several _years_ due to the reliability of the power.
This happened to an Open University researcher who stored several years worth of data on a local machine and had no backup that was remote from his/her office, a fire destroyed most of the equipment but the manufacturer managed to recover most (but not all) of the data from the fire-damaged disk - an expensive recovery process. Alas, I cannot find any record of this event online
I worked in the City at one point and whilst the parent company was huge, this office was a small offshoot that was trying to grow their business. However, in order to do so they needed to have two datacenters and the office all connected, along with dual ISP uplinks with AS failover etc.
Problem was, they never gave me any money for equipment or staff, I managed to scrape together all the components for a Cisco 12000 series chassis from the cupboard spares at one point.
Anyhoo, whilst testing the BGP failover I had to clear out the tables to get everything talking nicely (since the office wasn't connecting the two datacenters together at this point I was running the BGP peering from the two DC's across the internet in a GRE tunnel - not recommended practice and there are quirks to overcome, but it worked).
The MD went nuts when I cleared the BGP over a lunch-time break and told me never to do it again or I'd be fired (should have got out then really). I wouldn't do it after hours, because bang on 5.30pm all the people left in the office started smoking, and there were no opening windows! Nasty, and since I'd given up smoking I wasn't staying around for that.
In the end I simply set the routers to reboot in 10 minutes then I made sure I was talking to some sales droid within the MD's line of vision when it all went shit-shaped. I said I'd go and look into it, and lo and behold I had the network up and running again in a couple of minutes.
When asked what the problem was I told the MD that the BGP peering became saturated and de-stabilized the routers' back-plane memory, forcing a reboot, and that adding some AS path filters to strip out unnecessary advertisments would keep it under control :)
In the distant past I was providing support for an EDI supply chain system, the one I was currently working on was running one of the large distribution depots for Marks & Spencer, essentially everything went through here.
I talked their IT through tidying up a bunch of backup data files that were impacting performance. I said, and I'm pretty sure I said this right, "type Del *.0??" what she heard was "type DEL *.??"
Sadly all the current data files had a 2 letter extension and we were running this in the data directory.
All their data had gone. I asked about backups and she had the audacity to say 'I don't know about that I'm the Unix admin'. Turns out that they didn't keep backups.
Fortunately they had a functional spare system that we could replay all the changes into and we got them back up and running after only something like 12 hours of solid work.
Heh, had somebody delete a critical data file from Ulti-Sales (POS system I supported back in the day).
Poor lady didn't know what to do. Of course I tried a rebuild first. No luck. (It was support over a phone, luckily she was computer literate, and knew what to do).
Asked her about backups, and they did have it. I got her to restore the backups to a different directory, then copied the critical file over to the live POS system, and ran a rebuild - and all was well, no data lost.
She was happy. And so was I.
Many years ago one of my colleagues was writing a data submission process. This wasn't going to be for anything critical, so she was writing this as a simple bash shell script (AIX UNIX), which would be run from cron every 5 mins.
The data would be coming into a dir, via some network shares and internal FTPs (late 90s). With the data being submitted/pushed into an old ESB system we had.
The data was archived as it was processed (basically moved to a daily archive dir, with .hhmmssnn appended to the file name). The original file was to be deleted once it was confirmed it had been pushed into the ESB okay.
She set up some test data on the dev environment, and kicked off the script manually, and happily sat there monitoring the ESB frontend, as her test data came through, confirming each file, and its contents were as expected. (You can see where this is going!).
Then she started to see odd file names in the meta data, that she didn't recognise, with content that she didn't expect to see. Yup, once the new script had finished with the test files in her test dir, it worked it's way up and down the tree, into any parent and then subdirs it could find, submitting, archiving and deleting everything it it's path! Wouldn't have been much of an issue if under cron, as the user there was always very restricted, but she was logged in under a generic developer account, which was basically like having god mode back then.
So glad we had nightly backups!
Needless to say, a few things were changed after that day, with generic god mode accounts being locked down or banned, and a lot of changes on users and group settings etc.
"Is there anyone on board that knows how to fly a plane?"
Yeah, I was in that situation once. Naturally I did the right thing and stood up, shouldering the responsibility for getting everyone safely on the ground. I mean, bearing in mind the alternative, who amongst us wouldn't?
The stewardess didn't think to ask me about the full extent of my flight experience so no-one ever knew that it comprised just a few hours flying a PC-based Tornado GR3 simulator, where I'd never got beyond the first mission of laying a JP233 runway-denial weapon along the length of an Iraqi airfield in 1991. I mean, who would know that a 737 wouldn't handle the same at 40 feet and 270 knots? I certainly didn't.
Still, every crash you walk away from is really a landing, right? Right?
Back in the day when I was switching between Apricot and PC Compatibles I will admit that once or twice I went to format a Floppy on the former with Format A: forgetting that was the HD.
Thank heavens for Norton Undelete! As long as you caught it before it finished you could run that and rebuild the file system in a few minutes.
Did this with an entire HDD which I was suppose to migrate to a new one on my PC build. After 3 days I learnt the best way to save £30-£100 on file recovery software is to learn how to use TestDisk.
Thankfully it was a quick format, and I only had to copy back the (hidden) backup file partition, a process TestDisk finds and automates well.
Some twenty years ago a not too bright colleague deleted a file on his Irix workstation. He downloaded from a Russian website a program that was supposed to undelete files on a UNIX system, but in fact was malware. It took our service department several days to restore some order. Fortunately the damage was restricted to this one machine and we mocked our colleague with the story for a long time.
I too, have experienced that ice-cold feeling in my gut ... on realisation that the database connection on which I've just issued 'truncate table' is somehow connected to a production database, not the dev database that my database tool suggests that it is.
On that occasion, I was able to connect to a disaster recovery spare and recover the data I had just blown away in production, but that was a seriously adrenalyn-fuelled half hour followed by a lot of beer to relax when realisation settled in that I'd got away unnoticed with no permanent damage done.
My first IT job as a programmer on a UNIX system. Note... we didn't have a "test" environment, only "live" data. I'm writing a search utility for marketing and did some cut and pasting from another script to speed up the writing process with the intention (what the road to hell is paved with) of fixing the thing. Come in the next morning and the boss wants met to run what I have done so he can see my progress. Cue mad panic as suddenly all the data on the mainframe disappeared. Luckily, the backups were on site and only about an hours work was lost by the staff. I kept my job as did the manager though we both got reamed for it. The following week a "test" server appeared at our door. The same one the manager had been requesting for two years.
That you can easily do much more damage to a wider set of data in almost no time.
I remeber quite well when I had messed up several directories with thousands of files of old, older and newer versions, a huge set of files with the same name etc.
It was already late int he evening, when I hacked a small shell script that checked various directories for file duplicates.
And, as expected, I somewhat messed up the check ("test, "), which at the end kept all duplicates and deleted the single files that should stay untouched. Job well done.
Mines the one with the backup tapes..The ones with "tar: read error".
In a past life, I sought to make a meagre crust with some Linux consultancy. In a neighbouring office was the region's Netware man, who serviced quite a few regular clients and made a much better living than I did. But he considered it madness to try and develop custom software on Netware. On several occasions when his Clients had custom needs, he introduced a single Linux machine to their network, and gave me the job of setting it up to meet the client's needs.
I know of a co-op who worked for AMD who, in his last week, decided to transfer some nifty scripts he had written back to his college account. He wrote a perl script and set it running only to hear a yell from the support person. It seems that he had a link to the K5 floor plan & wire list in the directory he was transferring so his script merely started transferring the crown jewels back to his student account (he was legitimately working on these files).
The yell was caused by exceeding the maximum size of transfers and filling up one of the server disks. They had to get into his student account and delete some of the files.
You never know what will happen when you run a script that copies directories over the network.
Sometimes you need to bullshit your way out of unwanted jobs. Like the time I sat in the pitch session for writing a football game and pokerface on talked about the difficulties dealing with 22 players PER SIDE on the 8bit hardware of the time... Never found out if my boss worked out why the pitch failed ;)
When I used to fly small airplanes, I flew a heavily loaded (3 passengers + pilot) into an airport with a short runway and a hill at the the far end of the landing run. Near touchdown, I decided that my approach was not good enough, being a little too steep and fast, so I belatedly transitioned to a go-around and tried to fly back out in the direction of that steeply rising terrain. The geometry was - less than ideal. As the little plane struggled to climb above that ridge line, I struggled to find the sweet spot between not quite grazing the trees and plunging into them while gently banking away from the little mountain. All the while with the stall warning intermittently spitting out little hints of impending doom.
My front seat passenger turned to me and asked if there was a problem. I coolly replied, "No, not at all."
Biting the hand that feeds IT © 1998–2018