Millennium Buggery: When things that shouldn't be shut down, shut down Happy New Year! Because you'll all be feeling delicate, El Reg thought we’d ease your pain with some of our other readers' more technical errors – distraction is the best cure for embarrassment, we're sure. Both are from the time of the approaching Millennium Bug, and as reader "Frank" puts it, "the imminent EOTWAWKI". At the …
Put them both together and I've been there. Network professionals leaving a PFY instructions to refresh a network adapter that don't specify which of the adapters you do it to because to someone with 30 years experience the idea that a noob wouldn't get the right one becomes unthinkable.
Though what amazed me was how long it took to get someone qualified to push a button on the site.
Happy Blade Runner Year, one and all.
If you follow them scrupulously and something goes wrong, you've got your iron-clad excuse right there in your hand.
And the blame gets shifted to the guy/team who wrote the instructions, and to the manager(s) who approved the instructions.
Don't you find that there's suddenly a lot less yelling in those situations ?
To be honest, I normally qualify instructions that I think may be a bit dodgy with an "are you sure about this" and get the response to that in writing. You may well get instructions that are half arsed given by someone who hasn't a clue. In this situation you're the expert giving advice that someone may not have considered and you're saving yourself grief by asking the question.
In my experience, they always make sure to have included something vague or technical enough that whoever should realize that I did the thing requested properly doesn't understand that. Then it becomes my word against the person who wrote the instructions, who is usually above me and thus the important person who shall not be questioned.
For example, I was tasked with writing a particular program that would be pushed to many systems in the field, and needed different configurations for each such system. They let me write the spec after that, so I gave them a program that would do all the things requested and would be pushed out with a configuration file describing exactly which of those features they wanted on each system. But they were irritated by having to push two files. I was never told why this was a problem, but was told to fix it. After receiving a spec that described the following modification, I wrote another program that would take the configuration file and pack it and the program together, and there would be only one file at the end, then sent this to my manager. Unfortunately, he chose not to read the instructions and was confused that when he selected his configuration options and name of client, there was suddenly this new executable when he pressed the "make distribution" button. I ended up called to his manager's office for a lecture about how I didn't understand how this business works, and what was I thinking when I implemented this. My reference to the spec he wrote, my suggestion on implementation, and the message from the original manager saying "This sounds like a perfect solution to that problem. Go ahead and write it." were ignored because manager number two couldn't/didn't want to read and understand the spec.
Posting anonymous for the possibility that they read this. I really doubt they do.
Documentation rocks - particularly when dealing with directors who could be described as "marginally sane". I once specced out a very large elearning project and sent it to said director for sign-off. (This was something like 140 pages long). It duly came back, signed by his own fair hand. This was a multi-partner project, originally planned to work in pseudo-VR and developed by a third-party company. The great day comes for the first live demo and the director goes utterly ballistic - it simply doesn't do what he wanted. He barks at the whole room, "This wasn't in the specification!" Unfortunately I lost my temper a little bit and snapped back. "Yes it was, I wrote it and you signed it off, did you read it?" Silence... a long silence. My career was floating away in front of my eyes. Luckily he was mad, but fair, and back at base apologised for his outburst - he hadn't read it. This was the day I learnt to provide executive summaries of specifications, in PowerPoint, using a lot of very simple pictures and diagrams.
" I normally qualify instructions that I think may be a bit dodgy with an "are you sure about this"
I am "Jim" and I did check with my manager in the UK that that was what they wanted and was given the go-ahead.
We used to have test runs on the networks from time to time. Always late at night as the the business worked a 14 hour business day and always with a "live" system. The place only closed for one day a year, Christmas day, so there was no slack time in which to pick up the pieces if anything went wrong. So I knew the score and just used the list as an aide-mémoire to ensure that everything was done as proscribed. I had learned in a previous job of the benefit of having a record of what I had done in a job that had more than usual importance and I was damned sure that I had my back covered for this one.
After all the excitement had died down I and my two companions spent a peaceful night chatting and helping ourselves to the emergency rations laid in. Come 06:00 my relief turned up and I went off to bed a fair bit better off than usual as I had received 15 times my hourly rate for the nights work,
Happy days!
"And the blame gets shifted to the guy/team who wrote the instructions, and to the manager(s) who approved the instructions."
Given the context in Jim's case it sounds as if the entire thing came from a manager who thought that any bit if kit still running would burst into flames or self-destruct in some other way at midnight.
In those circumstances there's no reason to think that the writer knew there were routers or, if they did know, would be aware they shouldn't be shut down. It's what happens then authority outruns competence.
Isn't that usually the case ?
Because, if authority was defined by competence, 1) our entire economy would come crashing down and 2) most IT admins would suddenly find themselves with board-level revenue, while board members would find themselves manning the Helldesk - and failing all Target Performance Indicators miserably.
There's been a lot of these articles lately....and YbF used to offer cash for videotapes...
A little dull here this xmas, not even one good drunken* family overload breakdown rant, not one....
* I'm no help, there - alcohol forbidden me this year on health grounds.
I just love it when you have something like this, and they won't let you have scheduled downtime for such actions.
Though not in the £50k/hour area, I had a similar situation before Christmas... updates had been postponed to an internal system because "we can't take the system down". Updates postponed a second time because "X needs to do Y so you can't update yet", and so on.
There are only a few windows in the year where I can update without everyone whining, and every time "someone" different had to do "something" important and it was always something that I, or they, felt couldn't be left half-done when updates were applied (which included database schema updates of the databases they were using, etc.).
So it got to December. Where I had a three week holiday booked. And so I stated that the update NEEDS to happen. There was much wailing and gnashing of teeth but I got it scheduled in two days before I finished (I'm not an idiot! Never do updates on your last day in!). I got agreement from everyone, a bit of "Oh alright then", but it got scheduled in, and notified, and notified, and notified. And then a day before, someone kicked up a fuss because they realised that there was yet another "something" that couldn't ever possibly be done completely before or wait until after the updates. Of course, that's in their opinion.
Speeding up their schedule to get it done before the updates "wasn't an option" (despite the fact that they'd had MONTHS to do it in). Putting it off until after the updates "wasn't an option". Doing it while the updates were happening wasn't physically possible and certainly wasn't sensible.
I didn't even attend the meeting. I just sent an email saying "It's been postponed multiple times, while it wasn't critical, and now it's critical. I will be updating". There was uproar. Not least because someone then went and told *some* of the users that because of the updates the system would be down on the wrong day anyway (I suspected an attempt at sabotaging the update from happening). And obviously the other users had all been told that it would be down on the scheduled day.
It was then that the emails started getting personal, wondering why I was allowed to have holiday (because I'm human?!), why they were being taken over Christmas (because most of Christmas is compulsory holiday anyway, and there were never originally any updates scheduled for Christmas, and I wouldn't schedule updates over Christmas normally anyway? Literally triple/quadruple-postponements are the problem, not that your IT guy isn't working over Christmas when you do no business anyway), wondering why the works weren't scheduled in (they were), wondering why we had to schedule in downtime for updates at all (which would be a fair point, if it wasn't for the fact that the software in question can't be running on any client when you apply updates to the server, and all clients have to have the same version number as the database they use), wondering why we couldn't pick a "more appropriate time" (I did! Several times! And people not doing their job postponed it again each time!), wondering why "someone else" couldn't do the updates (erm... because it involved both Finance and HR data, so nobody else should be privy to that information, plus nobody else on staff is qualified to do so - or even close! - plus you don't listen to the guy who CAN do the updates, so you can literally go find a new IT guy if you let someone else touch that system / access / password).
Turned out... we did the updates... as originally scheduled... nobody was affected... I was there a few days to check it all went okay... everything was fine... nobody (not even the users) shouted. A lot of fuss over nothing.
And now I can look forward to a discussion about "Scheduling all future updates". Which I do anyway. But now it will always mean "making sure my original schedule lets me throw people off the system", not just polite announcements. Because my schedule takes into account far more than their last-minute "I haven't been doing my job" reasoning ever does. So I checked the calendar for 2019 for when the system can go down. It looks like I have maybe a day in a week in March. And a few windows in July. Let's hope that a) no critical update is required, say for the new financial year or Brexit, b) not one user "has to" need the system in those windows, c) when I schedule it, people don't suddenly remember that ultra-urgent thing that they've put off all year must be done in that day because they're too lazy to do it before then.
Putting off the updates for every whim basically now means that you don't get the option to put off the updates. Ever.
To say that making a fuss over such a non-event was counter-productive isn't the half of it.
Making and enforcing decisions like this is what top management is paid to do, not the likes of you and I.
Where the updates are to enable new functionality that department X needs and Y is objecting then get X & Y to agree a schedule between themselves. If they can't agree then it goes to the top team to tell them when it's going to happen.
Where the updates are initiated by IT because they're needed to patch some risk or move off some component that's reached maintenance EOL if you can't get agreement then go to the top team yourself and point out the risk and that you can't accept responsibility for any consequences of postponement.
I've seen the latter happen once when the server was due to reach EOL at the year end and IT proposed to cut over between Christmas and New Year when the business was closed. Management deferred for some weeks without giving reasons but accepted the risk of it costing arm and leg rates if there had to be a call-out after that. The reasons emerged early in the new year; it justified the risks for them.
Nobody who wasn't a competent techie should ever manage competent techies. Managers are secretaries, nothing more. Yet even great techies can make poor managers. Different skill sets, very small intersect.
I just quit jobs when my manager was replaced. A good manager was so rare, and an average manager so damaging, that it was better just to walk away. I'd guess at least 98% of tech managers suffer from imposter syndrome and 97% of tech managers are imposters.
You know your manager is good when you feel you've let them down and yet they haven't said anything. A bollocking from an ahole has no effect.
It's not just in techies either.
People can be promoted to strict rules orientated industries who then simply make up their own rules to go over the rules framework and wonder why people are aggressive about it.
Because the original rules were more likely NOT to kill a thousand people in a masterstroke.
Happens in all jobs. I used to be a D&T teacher, a 'head of year' came charging in one day disrupting my class for no good reason. When he rushed off to harass someone else one of the kids remarked, 'It's funny how Mr X had made a career out of walking fast and carrying a clipboard.'
managers are incompetent techies who got promoted to move them on from a role in which they were useless.
Reminds me of a quote I think in one of the Narnia books (or another CSL book) about a teacher who was so useless she had to be promoted to get her out of the classroom.
Or something like that.
It's in The Peter Principle, known as "percussive sublimation".
The principle itself is that in any hierarchical organisation, employees get promoted until they reach their level of incompetence. The funniest things are true, and the book is well worth reading.
"Nobody who wasn't a competent techie should ever manage competent techies. "
I know a lot of former competent techies who became fucking awful managers.
Managing is a vastly different skillset to teching - and requires training that usually isn't provided when the "promotion" is given. What's even worse is having an INcompetent techie promoted to manager (which happens far too often).
So much this.
> You know your manager is good when you feel you've let them down and yet they haven't said anything. A bollocking from an ahole has no effect.
I..... once killed DNS for one of our largest clients, for around 2 days, because of a change which I made, backed out (including stupidly removing the change notes for it, as it "hadn't happened") which occurred over a weekend, while a number of staff were on an entirely different continent.
I walked in Monday without a clue nor a care in the world, my team leader turned to me and told me I'd removed the DNS entry from the firewall, why would I have done this?. My face clearly said it all, as it was hardly brought up, until I started giving myself grief over it (and every time there's ever a DNS issue I immediately announce it wasn't me!).
Where the updates are initiated by IT because they're needed to patch some risk or move off some component that's reached maintenance EOL if you can't get agreement then go to the top team yourself and point out the risk and that you can't accept responsibility for any consequences of postponement.
... and when you do so, do make sure to include a printout to Equifax hack postmortem. Not a link - hard copy so they have no excuse for not knowing the dangers of delayed patching.
As usual, all this is down to the failure of management to actually manage. There is no excuse for postponing an update at the last minute ; if management had taken the need into account, it would have planned the workload accordingly and made the time available.
Unfortunately, management today has nothing to do with such mundane things as planning and everything to do with Cover Your Ass. Since nobody has the balls to actually impose a deadline, we're all stuck with waiting until we're back to the wall and then we can only pray that nothing will go wrong - which, given Murphy's Law, has a coin toss' chance of happening.
"As usual, all this is down to the failure of management to actually manage."
In fact, a failure to manage at two levels. Good management should create a cohesive workforce that doesn't get into the turf warfare which seems to be the essence of this particular incident. So not only is the management not planning on the task by task level, it's also not showing proper leadership.
Our Parliament looks like an interesting case of extreme management failure. Both for itself (failure to manage necessary building maintenance) and for the country (b*****).
While few can aspire to rival that, we can nevertheless use it as a yardstick. This never-never update could no doubt clock up a few MicroParliaments.
"There is no excuse for postponing an update at the last minute"
I've got BT nagging that they want to do a maintenance visit - on equipment that was installed in what's now the bosses office.
The equipment has been reliable. The visit is simply to verify the installation is clean and safe.
The boss is busy. BT are insistent. I've given them days when he's not in the office, but that doesn't suit them, they want days which suit them.
Which is ironic, because they phoned up and apologised for delays in replacing said equipment - 15 years ago - promising they'd get it sorted "in the next few months" (it's now pushing 30 years old and has failed a couple of times, repaired by cannibalising parts from scrapped kit)
It's now 191 months and counting since they promised to replace the installation real soon now. I wonder how many times they put it off?
I hear you....
I work under a manager who has to consider every single corner case, and, 9 times out of 10, some new feature won't be adopted because there's a chance Gerald, who works in bumfeck, alabama, won't like the colour green on a tuesday. Not quite the same, I'll grant you, but the feeling is the same.
Putting off the updates for every whim basically now means that you don't get the option to put off the updates. Ever.
If it's really as farcical as you say, get out. Get your CV all polished up and move on. They'll just keep expecting you to pull miracles out of your arse every time. I've been there. I've done it. You're totally overlooked and unappreciated if they overrule you each and every time you schedule work on your own time. That's just toxic, and you'll never change it. And when a problem inevitably happens, it'll all be your fault.
I've been that guy performing an in-place upgrade of Exchange Server on a Wednesday night because the boss didn't want to use a VPN to pick up his email, and neither did his mate in Dubai. I've been the guy reinstalling the Linux boxes over Christmas via the iLOs from home. I've been the guy dealing with one group of people moaning that the backups are interrupting their work at midnight and you can't possibly do work on the servers in the evening, and another group starting at 8am, 4 timezones ahead of you. Nobody cares, and the best you'll get is some arse going "I don't care what everyone says - you're alright. <arf arf>". Looking back on it, I should have been out there a lot sooner.
Fuck 'em. Go. Run. Only look back from a safe distance to watch the flames. Good luck!
Icon for ESCAPE!!
This isn't related to shutting everything down, but it is related to religiously following instructions. This happened in about 2008 - a customer of ours had some particularly tech-phobic staff and we got a call one morning from one of them saying that her email had "disappeared". She couldn't find the remote support icon either. I tried to talk her through starting Outlook, and she said it just says something about setting up your email account. They were local so I drove out to have a look. Sure enough, everything was wiped off their computer, and it looked like it had been factory reset.
I asked the lady what happened when she started up the computer in the morning. "Oh, nothing special" was the reply, "just the usual stuff when it starts". Then she added, "I did what it said". I began to get suspicious. I went through the startup procedure with her. The screen flashed up its usual "Press F11 for recovery" screen, and she interrupted "Yes, so I hit F11". I asked why and she said, "because it said to press F11". I pressed F11 it to see what it does. It came up with a screen asking if you want to continue with recovery to factory settings and to press Y to proceed with that. She said, "Oh yes, it told me to press Y so I did".
And so she actually told the computer to wipe its settings and data (even though it warned her that all data would be lost!) because "I thought that's what it wanted me to do. It said to press Y to wipe all my data".
I asked her if it had occurred to her that she shouldn't wipe everything, and all she had to say was that it shouldn't have told her to press F11 in the first place.
She left the job shortly after that and the company folded a few months later.
The screen flashed up its usual "Press F11 for recovery" screen
This is, of course, bad UI design. It tells the user nothing about this being an option for which there is an alternative, nothing about it not being the normal option and nothing about the circumstances in which you should use this option. Don't assume knowledge the user might not have.
It should say something along the lines of "If and only if you need to recover after a serious error press F11. This option should only be used by technically knowledgeable support personnel. If in doubt do NOT do this; just do nothing and the computer should start up normally and ask for help if it doesn't."
"This is, of course, bad UI design."
It's also poor setup - almost all BIOSes with this feature have the option to turn the message OFF
Many years ago, back when I ran an ISP and virus transfer by boot floppy was a common thing, I made a point of going into BIOS settings when onsite, showing the boot order to customers, explaining the danger of having it booting off floppy or cdrom and then switching it to prefer HDD boot (disabling floppy boot entirely where possible)
Most of them would make a note of this (I encouraged the to do so) in case they ever needed to boot off floppy/CDrom (it was W95-era - a little after the period where bootable games were common, so needing floppy/CD boot was uncommon except for recovery)
One of my customers was an exec for a local power company. As I was setting his machine up and finished going through the procedure (which included dropping a copy of f-prot on his system(*)) the company sent out an "IT expert" who walked him through various stuff.
She then gave him a lecture about viruses on floppies, etc and to always make sure floppies were ejected before rebooting. The look on her face when he commented that the ISP guy had just said the same thing and additionally disabled booting from floppy was priceless - she was adamant that such a thing was impossible and had to be shown the setting in BIOS. My customer told me that after I left she'd attempted to unset the boot order and reenable floppy boot so it conformed to "her" idea of how a BIOS should be setup. He asked us to drop over and recheck things the following day.
Our helpdesk was agnostic and we spent more time getting customers of other ISPs back online (for a fee) than our own(**). It was a nice little earner. Most of the local computer shops tended to "format and reinstall" so recovering lost media files was another sideline (and earned us a LOT of repeat business even though we freely gave away the tools to do this job).
(*) At that point F-prot was free for home users but I paid an annual fee to cover the installations performed anyway.
(**) Most "ISP installation disks" would stomp all over all existing settings and hardcoded everything. Some went as far as locking the DNS settings in several locations, which meant that if people used multiple ISPs the competitors would compare badly due to DNS latencies.
We showed customers how to undo that damage and how to use automatic settings or use per-ISP setups if they had multiple providers. I think the printed how-to sheets were the most appreciated part of the service.
I asked the lady what happened when she started up the computer in the morning. "Oh, nothing special" was the reply, "just the usual stuff when it starts".
I had one of those too few years (and beers) back. No backups of course, yet somehow it was my responsibility. Even though they'd never been a customer of mine before that day.
Even worse, they "had to do work on the computer" before bringing it in, despite being told that they must NOT use it if they want me to have any chance of recovering their data.
Somehow I was responsible for their lost data, despite not being the stupid twit who wiped things despite clear warnings that all data would be lost!
I need to go and lie down for a few hours.
I can still remember the old Foundry (then Brocade, then Dell) course in Bracknell. Never got to finish it due to a dislocated shoulder, but the inability of the router code to take any bloody notice of static or dynamic routing config changes in the labs without a reboot was notorious. Not to worry: it had a "fast reboot mode" which would only take 3 seconds! Well, that was a fix as far as Foundry were concerned :-)
Fortunately I learned the lesson of not messing with remote routers and firewalls by screwing up my home one from the outside. That was at least fixable by a phone call to my wife to tell her to power it off and on again. I hit the enter key, everything stopped and I immediately saw what I'd done. Oops.
Now I know to (1) never do that and (2) if it really must be done, schedule a reboot for five minutes' time before entering the command, on the basis that if I can still talk to the router I can cancel the reboot, and if I can't talk to it, hopefully it'll be back in five minutes.
You should never change parameters on a remote system unless there's someone local who can restart it or out of band remote control. Even if it's tested, it will work only 99 times out of 100.
Did a fairly simple change to a server. Told it to reboot, and waited, and waited... Drove half an hour into the office only to see a prompt that hadn't appeared before complaining about something trivial about SCSI. After that, made sure the DRAC card was working. The server never produced the warning again.
Yes, a remote reboot assumes that there's nothing wrong with the hardware that might prevent a reboot. For a simple router box that was reasonable, the only hardware concerned was required to be working to be able to talk to it in the first place. I guess if the system disk was well and truly fscked then it might decide it couldn't find critical files after reboot though, having trampled through the inodes with the CPU equivalent of hobnail boots.
One springs to me mind where my then boss emailed me telling me to install the Exchange 2010* Management Console onto a couple of Citrix servers
Then tried to blame me for not uninstalling the Exchange 5.5 management console from them.
Because I should have somehow known...
Of course I pointed out his failure to instruct != to my perceived failure to act and since the two consoles can be present on the same machine to manage both environments, how was I supposed to assume?
I can imagine the screaming if I'd taken it off without being told to find it was still needed.
*Might have been 2007. Was a good few years ago now.
Though at least I would have had someone at the remote site to call and fix things. Which was a good thing, because the Atlantic Ocean was between me and it. In my case I needed to add a VLAN for asynchronous storage replication between sites, which the network guys were supposed to have done but guess they didn't think it was important enough to actually work that ticket that had been in their queue for a week. Fortunately for me (and potentially unfortunately for them) I had the password to the network devices.
I had started to type in a command, then realized its successful execution would cause my connection to drop before I could type in the following commands. So I put them in a script that would execute off a server inside the network, spent a few minutes looking over it super carefully, then decided to comment out the "meat" and just do the 'up/down' portion with a sleep 5 between them as a test, and only when that worked did I decide it was safe to do the whole thing.
We have a special file drawer at work where we file very odd things that are just too unique to go into the round file. One of them is folder that contains quite a few letters asking for us to prove we are y2k complaint. I wonder if I should reply to them yet.
Yep, I was thinking that too. My Very First Home Internet Connection (apart from dialup to the office) - except when the aerial fell off the chimney (not T2's fault: the bricks were rotten). But I only got 128k--. Very dependent on where you lived, as it needed line of sight to the base station, which was on a tower block in the centre of Reading.
> "Hang on," said Frank, and, because he had the terminal window open at the time, he typed in the command to disable OSPF... on a router to which he was connected remotely.
Oo, ouch, I've done that. SSH over TCP but same principle. Just started on a SysAd contract having only done it as part of building something major vs as job-focus and Production, didn't have the proper anal-focus mindset, needed to restart TCPIP for new config, typed the shutdown and restart commands as 2 separate commands.
Or at least, I started to...
Oddly, there was no response on the pseudo-remote ssh screen when I tried to type in command #2.
Baffled. Confused. The penny dropped. Argh!
Fortunately, it was just a Xen VM (Xen kicks arse, btw. Something like a 3 or 4% penalty vs bare metal? WTF!?), and the provided VNC API gives Console access. Back online. But ... whoops.
> "Hang on," said Frank, and, because he had the terminal window open at the time, he typed in the command to disable OSPF... on a router to which he was connected remotely.
When working at a US military base abroad, our higher echelon was the only level permitted to work on our firewall or tier 1 router. With predictable results, twice per year.
Sudden service outage.
Telephone rings, sheepish voice from a two hour flight away notifying us that he banjaxed the router/firewall configuration and he'd walk us through the steps to break into the otherwise centrally (from their site, of course) access controlled setup.
No need, with having to do this every other quarter, we could break into our own equipment drunk, sleep deprived and kicked in the head by a camel. Which is precisely what we thought of the intelligence of the caller.
Besides, we already hard coded our logons into the equipment the first time, they entirely missed that fact and we fixed the idiot's error far faster than it would have taken to bypass loading the configuration. We also had taken the liberty of connecting a console cable up to a well secured, highly specific server that those idiots lacked access to and never had to even bother going near the service room.
Although, we were seriously tempted to block all traffic on SSH from their network segment...
Instead, I made certain to mention the incident in our theater wide information security teleconference and our weekly Command and Staff meeting, so that the General got a good chuckle. Along with, well, everyone that was on the line. Some, coworkers of said rocket scientist.
All remembered, before I went into information assurance, I was our installation's BOFH.
As for The Princess clients, we've all had them. All suddenly have a Special Project that absolutely cannot be interrupted for even a microsecond, when it comes time for an assigned service outage for maintenance and patching.
"Absolutely impossible for this quarter, we're in the midst of a major effort here and are busy 24/7!"
"Oh? Well, your services and systems were patched yesterday, while the lot of you were at lunch and your systems globally idle. That's why your systems weren't locked, but had to be logged into when you came back from lunch."
Only had to intervene with three systems and extend the service timeout a bit on the server, as the "enhancement" made the blasted thing a bit slow on starting.
At least, that's what my script e-mailed me to report after patching and our hell desk never had so much as a blip from The Princesses.
Long ago I was working with a colleague in Telehouse, for a large ISP which failed after the dotcom bust.
We were trying to patch one of our servers into one of the switch ports off the core, and were having trouble tracing cables through the rats nest at bottom of the rack. Our method was to gently tug each cable at one end and the other colleague would feel the tension. All was going well until suddenly the entire rack of core routers went dark and suddenly the noise in the small suite dropped alarmingly!
We discovered that the PDU for the rack had a simple mains switch, without any shrouding, and a mains cable was tight across the PDU. Murphy's law prevailed, and the tight cable cut the power. I quickly powered the rack back up, and rang the NOC. Then I re-routed the mains cable.
I thought we'd get a serious bollocking, so we prepared ourselves for that, and an explanation of the cabling mess. Fortunately, the network operations people admitted the cabling was a disaster waiting to happen, so we didn't get into too much trouble.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
