A hamfisted worker at colo provider Telecity shut down "several" ISPs and their customers across the country when he started pulling plugs at one of its datacentres late last night. The hapless wire-scrapper then proceeded to make matters worse, by trying to fix the mess himself. El Reg hasn't been able to discover the …
we had a new security guy come in. Big company, huge patch system. He did an audit of the patch room. Came back after a few hours with a bag containing all the cable labels, saying labeling the cables was a security risk. The look on his managers face was a peach
Why weren't the cables labelled? Every cable in a datacentre should say where it comes from and where it goes to at each end (including port numbers). Cables are designed to be unplugged and plugged back in, it is therefore pitifully obvious that this will happen at some point, and if you have systems that are port specific you need to know you can unplug everything and plug it back in with no probs at all. As a storage person, I would be pretty pissed off if I had to replace a director card and found that none of the fibres had been labelled.
It may be the fault of the person who unplugged the cables, but the extended outage seems to be the fault of the installation engineers and the processes that they work under.
Indeed, this has Telecity written all over it!
Mine's the one with the prod in the pocket... (I'm going monkey hunting!)
Why am I not surprised?
I have had personal experience of This Place and their lackadaisical attitude to client kit - in my case, it was tape stackers that they....
..actually, I can't go into detail, my blood pressure can't take it.
My opinions on that particular visit from the F***-Up Fairy can be found here:
It's time for my therapy session, I'm still trying to handle the memories and the repressed rage!
(Paris coz even she could do a better job!)
Boneheaded Move Here Too
A good number of years back, working in a company which ahd a number of switches, but hadn't sprung for any of the bonus features, like backplane links or anything.
So all the switches were connected with patch leads (I Know, I know bandwidth bottlenecks and what have you)
Anyway, someone, it may be me, it may not, was 'tidying' up some cabling, and ran a new patch lead between two switches......
Two switches, that were already patched together....
Oh the joy, the joy...
Traffic on the network, fairly rapidly ground to a halt.
When the culprit cable was identified, the person responsible was of course on the wrong end of many jokes, and the cable itself was mounted on a piece of wood, and hung above the server room door as a reminder.
Anonymous, as I do have some professional pride!
It's such a pity...
...that the ijeet did *not* plug the live wire into where the sun don't shine !! That would have proved Darwin's case for the improvement of the gene-pool and the survival of the species !!
At least you know
It wasn't a BT engineer as this "Idiot" managed to at least re connect cables to ports ;-)
is this /.?
Some of the comments about this retard appear to have come from retards themselves.
Feck off back to /. will ya!
Because fact into doubt won't go...
So, looking at the story and originating blog: some guy writes that some guy says that some other guy is an idiot.
It's mostly fact free and yet were happy to point and laugh at some poor f'ker, who presumably will know about the incident's reporting on the Reg.
Cables should be labelled. People who are working on live kit should be qualified and have some sort of plan or process for the work they are doing. If this was not the case here it's a failure in managing the datacentre.
Re; Is this /.?
I take it you've never made a mistake in your life then?
'My' story is of an Electrician (I was a Computer Operator at the time, when computers were real computers and not these dinky little beige boxes - ah, George 3!) who had to change a lightbulb in the Data Centre (only we called 'em Computer Rooms at the time)...
Poor chap had had to wait a couple of minutes when he arrived as the 'front door' to the computer room had keycard access control on it and we were kinda busy. One of my colleagues let him in and he changed the bulb, then headed for the door. Since we were all still working, our little electrician friend decided to see himself out.
Darn, forgot to mention that the computer room had a raised floor, with a ramp from the 'front door' to an inner door, bit like an airlock if you will. So Alfie gets to the inner door and in a rather loud voice says, "Which button do I press to get out? Is it this red one?" as he reaches for the EPO...
Y'know that bullet time effect from Matrix? It was like that - visuals slowed down, sound seemed to drop several octaves and become a long, drawn-out shouts of "N-o-o-o-o-o-!" and half-a-dozen Ops start heading towards one hapless electrician but we were too late - he pressed the button and...
Nothing happened. Not a sodding thing. Well, the button went click and six grown men collapsed on a skinny bloke in blue overalls but nothing really significant.
Our much-vaunted EPO, designed to kill ALL power to the computers in case one of us came into contact with a live (415volt, lots of Amps, 'chargrilled Op in 30 seconds') wire, did ab-so-lutely nothing. Nada. Diddly-squat.
Ah, but there was some wailing and gnashing of teeth over *that* one, let me tell you!
I can have a little sympathy for the guy. Many times I've ripped out code that clearly could not work, has no references in the main codebase, and was commented everywhere with "Doesn't work." Of course the next day is catastrophic. Some obscure system that used to spit out errors all day now crashes. The obscure system is still listed as a health check for the hardware that replaced it two years ago. The health check failures cascade and everything is dead. Now comes 100 "WTF happened" e-mails, followed by a dozen meetings on the same topic, followed by a new set of rules so specific that they have no general value for future events whatsoever.
Okay, Thursday's 1650Z anonymous -- are you seriously telling us nobody's ever made a CLM on a border router's BGP configuration?
Knowing Datahop and their attention to detail - they would have been but if it was just a random member of Teleshity's 'intelligent hands' staff that was doing something - im surprised they could even read the work request properly.
We had a feed from DH a while ago for over 2 years - never had any issues with it. On the other hand, Telecity constantly screwed simple requests up from rebooting a wrong machine (we send request for server 123, and they reboot 124!?) to removing the wrong hot swap drive even when given the exact label.
At least everything is back up now.
What I did once
At my last place of employment (an electronics factory), I turned off a flow soldering machine because none of the isolators in the room where I had been working late sorting out someone else's foul-ups were labelled. Those things take a full day to heat up to working temperature. I only kept my job because of how much s#!t the company would have been in if I did the usual "disgruntled former employee" act. (And nobody else would have done my job for what they were paying me.)
And a former colleague of mine once managed to write a WHERE clause that matched every record in the database, not just the one that he wanted to alter. This was in an old version of MySQL that didn't support transactions, but fortunately most of the information in the database was duplicated elsewhere -- in disparate files. It took some amazing Perl coding to fix it. Since then, he always added LIMIT 1 to the end of his queries; and I test out my WHERE clauses by first using them in a SELECT and seeing if the result set is sane. If anybody asks, it's to cache the matches in RAM and so speed up the real query.
The tale of the falling tape ring
Well apart from the cleaner pressing the data center EPO button, my favourite story goes back to the days of IBM Mainframe consoles, hardwired keyboards and screens with a cluster of keys next to the main keyboard, one of which would halt the processor, not a problem in it self, but putting it next to the tape machines, where real people work was the problem.
...... Long long time ago when backups and processing was performed to and from tape, big spools of 1/2" tape, the write enable mechanism was a plastic ring in the back of the tape spool. Good practice was removing the ring when taking it off the tape machine, but what to do with it, well throwing it into a box was the usual practice, but during night shift, just sometimes you might throw it high into the air, say 20 or 30 feet and let it fall into the box, if not, collect them all up at the end of your session.
Yes that's it ... (a million to one chance) the falling tape ring managed to hit the stop button, bringing the mighty mainframe to a crunching halt. Hours of delays and lots of explaining to do. Paul.that did make the rest of us laugh.
The fix well a small clear plastic box to cover the keys.
I'd say the good old days, but really I did hate night shift.
'Fire Incident' @ Telecity HEX
Now I've been told that they've had a 'fire incident' at Telecity HEX last night - someone burning the CCTV evidence of their previous crimes...?
Try harder next time...
I once did a planned work that took me 10min instead of the usual <2min, when I explained to the boss that I had been a bit thick in not noticing a "transmit inhibit" had enabled after I changed some settings (neglecting to mention the pub dinner before shift start) he said: "That's why planned work notices are scheduled for 10min even if it's a two minute job."
As my lecturer in broadcast engineering said to us "Your not a real engineer until you've taken a broadcast network off-air." .... I'm a real engineer several times over.
Paris, because she knows where everything goes.
Big_Boomer your logic genuinely scares me. I don't understand how people like you and the plug-puller have the brain power to walk, let alone do anything with computers (like say, turn them on, or pull out their plugs..)
The worst thing about this story is that once again, one person can be blamed for a MAJOR problem. But it's not really fair to blame him alone since he shouldn't have been allowed in that situation in the first place. The whole system is f****d up. Quite scary really. What next ?
PS - What does "I know I have done a bonehead before but not for a while now" actually mean ??
PPS - Are you sure?