A plumber with a blowtorch is the enemy of the data centre Welcome again to On-Call, our regular week-ender in which readers share their tales of possibly-career-ending errors. This week, reader “Harry” tells us he's done support for the same global company for the last thirteen years and in one role found himself doing network support. One cold, cold winter's day, that gig took him …
Apparently where "Harry" works, they don't have building codes.
Where I live/work, it's a violation of building electrical codes to run copper data cables between buildings, due to both power ground loop issues and lightening issues. All data cables run between buildings have to be optical fiber (or wireless connection). Even between trailers parked next to each other.
This post has been deleted by its author
This post has been deleted by its author
Here in the UK, most of the internet to homes and businesses is by cable and twisted pairs. These pairs and cables go from one building to another!
Twisted pair is OK since it's isolated from ground, but running co-ax between buildings on different mains supplies has problems, especially if they don't have a common earth/neutral wire. The difference in earth potential might only be 2-3 volts, but a few hundred metres of co-ax might have 0.01ohm or so of resistance via the sheath. 2v across 0.01 ohms leaves 200A flowing in the cable sheath, which will melt most co-ax.
Oh look, a cable with a connector.
Oh look, a box with blinkenlights and sockets.
Perhaps they fit...
Perhaps they should be connected ...
Why is the internet broken?
Lessons learned:
1) Use "proper" managed switches with loop protection, even at the edge.
1.1) No desktop switches.
1.2) Keep the switches locked up.
2) Monitoring - SNMP is your friend.
> 1.1) No desktop switches.
On a lot of managed switches you can limit the number of clients to _1_ - that prevents unauthorised desktop switches or APs from working.
If you've got a radius-controlled setup (packetfence, etc) then you can enforce 802.1x or lockout unauthorised MACs (this doesn't protect against MAC spoofing, but not all kit supports 802.1x)
I *still* get cabling engineers etc. who talk to me about network loops on large networks.
I have to educate them all the time.
I have had to say "You run the cables, I will take responsibility for the networking 'loops'" to at least one cable-runner who thought they knew better.
Seriously, who the hell DOESN'T run spanning-tree nowadays? Hands up? You're idiots.
And then when you try to explain LACP where you WANT several cables or fibres all going between the same two switches or devices, you just have to give up and say "Look, just do it."
Who doesn't use LACP on their main backbone with all their spare fibre cores? Hands up? Guess what...
(a) Redundancy (great when you need to repatch a cabinet without actually taking it offline - just disconnect only one patch lead at a time on the uplinks).
(b) Accidents.
(c) Bandwidth increase.
(d) Just about any managed switch in the last 15 years supports it just fine.
"Seriously, who the hell DOESN'T run spanning-tree nowadays? Hands up? You're idiots."
Spanning tree was (and is) great for what it's intended to do, but the stories of spanning tree storms are legion as networks grow well beyond their original designs.
TRILL is cheap enough that if you need mission-critical networking or exceeding 6 spans it should be included automatically in your core - and it has the nice side effect that you no longer need to mess with LACP between the switches.
But if you'd ran another cable, any other cable, even in a circle around the site?
Then one cable cut wouldn't be capable of cutting you off, and "network loops" can't happen on modern systems.
Wire your sites in circle topologies and it takes two independent cuts of critical cables - plus all their spares, extra fibre cores and parallel cables - in order to actually stop everything talking to everything.
Think that costs more than necessary? Then consider if you're using VoIP, your network is how people dial 999. And if you have any kind of linear or star topology, it's easy to run additional connections between existing switches that aren't directly connected to get the same effect.
"But my cable was cut" isn't an answer. Always have a backup.
You put in redundant disks, redundant servers, redundant storage, etc. And then you put one cable between those systems and the people who need them? Put in a redundant cable. And if you use it properly it will boost inter-switch bandwidth under normal operation (e.g. LACP) or it will provide alternate routing in the event of a cable cut (STP/RSTP).
A University in Hampshire has gone fully VoIP, and with one swift stroke lost their entire safety net, even the highest risk labs are VoIP only, the trades unions have asked for hard wired twisted pair safety phones that won't fail because someone cuts a cable with a digger and kills the entire network (yes it is that bad), but moving the data centre off site was more important.
Then there's the Trend 'engineer' who took down an entire microelectronics building by plugging a lead into a switch in a Trend outstation, and they didn't even know he was onsite.
As for stupidly running cables, the welsh electrical wizards working for a contractor, I'll call them Bloody Liars Limited, ignored all the planned raceways above a new clean room and used the shortest routes, the service walkways, so now servicing the hepa filters is all but impossible, access in some parts is by flat down belly crawl, but they saved loads of cable for spidge... And the maths prof dvc in charge of the project hailed it as a great success, the explosive plumbing is another issue.
"risk labs are VoIP only, the trades unions have asked for hard wired twisted pair safety phones that won't fail because someone cuts a cable with a digger and kills the entire network"
And when it's all resolved, the money allocated and the work orders raised, someone will think it's cheaper and easier to put the POTS wire down the same ducts as everything else. For extra bonus points, break the network while putting the POTS in.
About 10 years ago I got a call from a customer I looked after, the guy on the phone was complaining his Internet connection wasn't working (major panic it must have cut his porn feed).
I couldn't ping the site, not even the BT router.
So I jumped in the car and drove down to take a look.
The door next to the guy with the missing link was the one into the data centre, I peered through the window, nothing, I mean nother, there was nothing in the data centre, it had all gone!
Err John where's everything gone?
Oh they came and took it all away yesterday.
Well why do you think your Internet connection doesn't work?
BT took all of their kit away ? Did somebody not pay the bill or something ?!
Getting BT to haul away their old crap usually means having to dump it on them next time they turn up to fix something, and it's usually followed by howls of "Awwwww, do I have to ?"
OK, on closer inspection the computer room wasn't quite empty.
The BT router was leaning against the wall, it wasn't connected to anything and the cable had just been pulled physically out of the connector.
There was also a VA disk array which presumably no one wanted enough to pay the shipping costs.
The "they" were the people who owned all the kit, "the bosses from the US"
I suspect that the John was part of the reason they'd taken all the kit away.
(PS. Oh bo!!065s the spelling checker in this copy of Firefox is F*&^ed)
Had this once.
Builder was replacing flat roof while I was on holiday.
Came back home and found wifi wasn't giving any traffic.
Looked out the bathroom window to find the Ethernet cable from garage to loft had been blow-torched.
IP65 box and punch-down junction box solved the problem without having to re-run the cable.
One of my mates is doing support on one of the higher tiers of an infrastructure provider - they guys 'n' gals you talk to if you are $big_customer (and quite up from the 'restart the machine' ..."help"). His favourite is when whole segments go dark - because copper thieves pinch the fibre. Happened quite a lot a few years back when copper prices were way up.
Copper thieves also stole piping out of my university. Unfortunately this was connect to the He supplies (we have a closed system to re-liquify the stuff for cooling purposes). Damage in copper was low. We lost a lot of He (which is *f'iing expensive*).
(icon 'cause that's how they try to extract the copper from fibre cables)
Quite a few years back when doing some surveying work for Nynex I recall a large fibre cable being cut, as it was explained to me workmen in the building, had encountered it as an obstruction, a large cable, that was apparently armoured and buried with extra protective sheathing. Their solution to the protection was a bigger saw.
(and quite up from the 'restart the machine' ..."help")...
I have dealt with these types of provider and I have been asked if it would be lot of trouble to reboot mainframe of a multinational merchant bank in order to test a an out of country WAN link really wasn't working...
I suggested this was not a good idea and it would be great if perhaps one of the engineers could take a look...
I have not noted much difference in the approach of $big_provider in my experience...
I once made the mistake of trying to explain to someone that you don't even "boot" a mainframe, you execute an IPL. And anyway, that isn't going to help with what you are describing as an OS problem as the thing has to run multiple OSs in multiple LPARs just to actually turn on properly.
"But that makes no sense, it has to be the hypervisor, just reboot the thing".
ಠ_ಠ
"Unfortunately this was connect to the He supplies (we have a closed system to re-liquify the stuff for cooling purposes). Damage in copper was low. We lost a lot of He"
Too bad you can't have a continuously circulating (or just capped) set of pipes containing hydrazine running alongside.
(Who me? Evil death dealing b'stard?)
Back in the late 90's I was hopping around Europe evaluating DC's to extend our network to and encountered the most bizarre setups you could imagine, one place even had a swimming pool on the roof that was plumbed into the cooling system - genius :)
However, this wasn't the place that flooded. That place was the one where we had put some pilot kit and were prepping the whole place with raised flooring etc. when we got a full on DC outage alert - including power.
When we inquired as to the cause we were told the place had flooded, like it was under a foot of water! This was a surprise because the place was fairly high up compared to the surrounding area.
Turns out that a moronic truck driver had driven into the corner of the building, causing a small section of it to collapse. Along the way he had also managed to take the head off a water point (the kind fire engines to connect to) which was now merrily spouting water upwards and then being diverted into the newly opened hole in the wall by the underside of the truck, the front of which was lodged 3 feet inside the DC!
It was so bizarre I was convinced they were winding me up, but they sent pictures to prove it :)
" Along the way he had also managed to take the head off a water point"
This has been frequently cited to me by fire brigades and various civil engineers as an outstanding reason why water points should always be under a pavement access plate.
Of course if one was to do this in the USA, dog bladders would explode.
This has been frequently cited to me by fire brigades and various civil engineers as an outstanding reason why water points should always be under a pavement access plate.Of course if one was to do this in the USA, dog bladders would explode.
Not just that, Think of all the Hollywood car chases that would be ruined!
With any change, one must assess the all-important unintended consequences.
We once dealt with a flood where the period of the waves in a 200,000 litre water tank managed to exactly line up, so that enough water would disappear down the other end to cause the float-valve to open - and that slug of incoming water nicely added to the returning wave in a lovely feedback loop. All until the many tonnes of water hit the end of the tank so hard it simply flew off across the room. The basement suddenly became a very wet place indeed...
"...why water points should always be under a pavement access plate.
Of course if one was to do this in the USA, dog bladders would explode."
That nut on top of a US style fire hydrant connects to a shaft extending down to a valve that's mounted in the (well buried) street water line. Except in the movies, a vehicle running over a hydrant rarely causes a gusher. Although, I did see one fairly pulled out of the ground by a fire truck that ran over it slowly.
Where I live, we have a mixture or ground-level and above-ground stand-pipes. Both constructed in the usual way, with a ball valve underground on the main pipe. The ONLY reason I ever see a gusher is a vehicle running over a hydrant. Invariably it is replaced with a ground-level connection.
I also sometimes see cracked pipe-- typically an even greater water flow, but not fountaining up into the air.
A Remote Job Entry device was temporarily installed on the floor above the computer room comms unit. The connection was full-duplex using 2400bps over two lengths of domestic twin flat cable. One day it stopped working - although the lights on the RJE showed it was being polled and was apparently replying. This was in the days before network monitors - so rather primitive diagnostics in the mainframe established that the RJE's replies were not reaching the comms unit.
The pair of twin flat cables came into the computer room via a hole in the ceiling - then trailed some distance across the floor to the comms cabinet. It was eventually spotted that the cables suddenly went under a floor tile - and reappeared on its other side.
The false floor construction was unusual as it was a matrix of steel bars that supported the tiles along all four edges. The floor tiles also had an underside of sheet steel.
Someone had obviously lifted the floor tile and then ignored the wires they had trapped when replacing it. To get the tile flush they would have had to jump on it very hard. The steel bar and sheet then acted as a guillotine - shearing the return signal's cable.
Two examples, both from the same place.
First. Wired corridor in a Polytechnic for Acorn Econet (bussed RS422 serial network used for BBC Micro's). One day, the network stops. We check each of the access ports (very basic 5 pin DIN connector in a box soldered directly to the wires in the cable to reduce contact resistance in connectors). All OK. Terminator, OK. Meter across the wires shows a short. Eventually tracked it down to a staple carelessly driven through the cable to 'tidy it up'.
Secondly. Camtec X.29 PAD used as an RS-232 terminal switch. In order to get it's attention from the PDP-11, it was necessary to generate a communication break (data line connected to ground for a second or so). DZ-11 or the comms software (can't remember which) could not do this, so I created an interposer that consisted of a 25 pin male D-shell connected to a 25 pin female D-shell with soldered wires between the pins, and the two held together with long bolts with several nuts holding everything in place, and a press-to-make, release-to-to-break switch between pin 7 and pin 2 (or was it 3). One day, PDP-11 gets slower and slower, and eventually stops. Reboot, everything OK for a while, then the same thing happens. Looking in the log, it was reporting data over-runs on one of the DZ-11 ports.
Turns out some vibration had loosened the nuts, one of the D-Shells had moved, causing a short from pin 2 to pin 3 (data out and data in or vice-verca) in the wiring to the press switch. Login banner sent from PDP-11 came back as if typed from the terminal, which generated errors and a new login banner. Eventually system was so busy fielding exponential amounts of data that it ground to a halt.
Moral. Good wiring is important, good soldering equally so.
Slightly OT, sorry.
Wired corridor in a Polytechnic for Acorn Econet
Had a room full of Econet at the Poly I attended in the late 1980s(*). Place was mostly VMS but had a few rooms with alternatives in - some XTs, some '286es (optical mice on optical mats!), some Apollos, and a room in the maths department full of Archimedes.
The Econet was in the engineering block, and was used to teach 6809 programming. Acorn buffs may be wondering how, given that the BBC Micro was 6502 and although there were several second-processor units available, the 6809 was not one of them (as far as I'm aware).
The lecturer in question had built his own second processor. Full marks for initiative, but the power supplies in the second processor units were so flaky that if you had to - erm - "hard reset" one of them for any reason, you had to make sure everyone else in the room had saved their work first. I'm not entirely sure how, but resetting a second processor unit would at the very least "take out" the Econet segment (usually meaning that people couldn't reach the file server any more), and would often (presumably via a mains spike) cause other 6809s to freeze.
It wasn't a heavily-used lab, and it paid to get friendly with a technician who could let you in when it wasn't being used so that you had the place to yourself. Not that I ever did.
M.
(*)When said Poly converted to a university after I left, the sign on the nearest dual carriageway read - for a couple of years - "University of Glam". I wish I'd taken a photo.
"[...] the sign on the nearest dual carriageway read - for a couple of years - "University of Glam". I wish I'd taken a photo."
When the ICL brand name was finally taken out of service all the old buildings' external signs were changed. During the change there was an ICL sign on the ground - alongside a neat vertical stack of individual letters ready to form "Fujitsu".
At little later it was noticed that the letter "F" was now on its own on the ground - exposing the "U" at the top of the pile. All in a neat row reading from above - "F" "U" "ICL".
"Turns out some vibration had loosened the nuts"
A video terminal on a naval air base kept losing its sessions intermittently. It was connected to the network by "thin" Ethernet. It was eventually found that there was a poor connection inside the wall socket.
That sounds simple - but the vibration had to be quite excessive to produce the fault. Someone finally twigged that it only happened when a Harrier jet was doing a vertical take-off/landing on the concrete pad outside that building.
"An up vote is yours for the mention of RJE."
The aforementioned RJE terminal was being tested preparatory to being installed in an office about a mile away. It was intended to replace the existing system - where a man on a bicycle with a traditional big basket on the front would ferry cards and printout.
It eventually transpired that the man on his bicycle was usually quicker than this latest bit of IT kit.
The fault lies with the 'plumber' as he would have smelt the burning plastic and should have stopped immediately and informed someone of his mistake.
He should have also checked beforehand of what other services are in the pre-fab, he could have damaged mains cables as well and burnt the whole place down.
Most good plumbers will have a fibre glass composite heat resistant pad with them to stop damage to wooden beams and electrical cables close to the places they need to use a blow torch, so the company unfreezing the pipes should get a bill for the damage they've caused.
A stern warning from HSE about putting lives and property in danger, I've worked in a couple pre-fabs using free standing gas heaters to keep the place warm!
If you've ever seen a 15Kg gas bottle go bang you'll know what I mean.
Back in the 80s a shopping centre near me had a plumber from a small local company in doing some work, with a blowtorch. He put the blowtorch down on a box for a moment, when he turned to retrieve it the box (full of teabags) was smouldering.
By the time he got back with a fire extinguisher the storeroom was on fire, and by the time the fire brigade arrived the whole supermarket end of the centre was burning.
A local garage had several new cars on display in the main mall, and there was some great TV footage on the local news of a reporter asking the garage boss if hs cars were still OK. She had no sooner finished asking the question when the roof of the centre caved in on the mall, leaving the garage owner with his "I'm on TV, I have to grin and bear it" face on.
Needless to say the plumber's liability insurance was a tad too small to pay for rebuilding a shopping centre...
Last job insisted on using an ordinary electrician for data cabling and telephony - it's just another bunch of wires, right ?
Most of it worked, some didn't, but hey he was cheap. For one particular lab I called in an engineer I'd used on previous gigs, to show them what PROPER Cat5E looked like and behaved like. Yes Chris, we like the pretty graphs from his test gear, but he's too expensive, limp along as you've done before. Grr.
For an encore, fibre runs were done by a sub-contractor of the electrician. Bog-standard unarmoured fibre between buildings, and ST cables tested by the good old-fashioned method of using a barrel connector and a lamp. The local rats liked to vary their diet, and the coating on the fibres hit the spot nicely.
Eventually, the whole lot had to be replaced. All 7.5Km of it.
I've had similar, an apartment block with Ethernet and CCTV cabling done with eight core alarm wire back to a central control room.
Some of it joined with screw terminal choc block connectors because it was longer than the 100M rolls the 'installer' had bought.
Amazingly it sort of worked, the main reason I was called in was because the pictures on the CCTV were terrible and he'd not installed the required number of cameras, instead he'd taken the money and run, the job had been 'quality inspected' and signed off by his mate who had taken a brown envelope filled with pound notes.
Took me a week and a half to sort it all out but it did pay for a couple of nice holidays.
Ouch, back when I did optical engineering the test rigs we used had jigs for wrapping the fibre around to ensure it maintained a minimum radius. Coil it too much and all of a sudden the light stops coming out the end. Not because the glass inside the fibre has broken, but simply because the radius of the fibre reaches a point where internal refraction effectively stops it dead.
Underfloor moisture sensor chirps.
DC staff inspect, under chiller transfer unit.
(we're over cap, as one should) turn off transfer unit, crank the manual valves closed, call service.
Service shows up, checks out the unit, finds tiny leak on input side of the chiller. Right down on the bottom. Under the floor line. Inches away from the sensor bundle attached to the bottom edge of the chiller.
Goes, gets torch, solder etc. Forgets a) buckets b) to inform DC staff that he's about to fire up a torch in a Tier rated DC.
Five minutes later, fire alarms, and the power in the area safeties off, and huge puddle of water under the floor to boot.
Only hit about 20% of the DC, but that only made the cleanup (IT level) a ton worse.
I don't know how it is in the UK be here state side the sparky union has a specialty for communication and data cabling. No I would not hire a regular union sparky to do cabling. I've seen the work those union cabling guys do. Cut to length, color coordinate, OCD cable management and labeling the ends.
And sometimes by street.
In the EU the unions don't have the same kind of stranglehold, however for the most part the primary contractor hires subcontractors who actually know a little bit about the data cabling.
I encounter a lot of aluminium "Cat 6", however for the most part the subbies do a reasonable job because they don't get paid otherwise.
Re Alu cat 6... yeah, I've seen that too (I'm actually a sparky but trained in the 80s on data and telecomms so I have what approaches a clue with data wiring, which means everyone thinks I can do it just as well for half the price. The bill often surprises them).
It's becoming VERY hard to get comms wiring (cat5/cat6/CW1308/CoAx) that's actually of a consistent standard, as most wholesalers buy on price and take the label as gospel.
For handy reference, a cig lighter will melt 24 gauge alu wire in a very predictable way, and a magnet will ID the EVIL copper plated steel variety
> I encounter a lot of aluminium "Cat 6"
The scam these days is "copper clad aluminium" - which is as it says. All the disadvantages of aluminium with no easy well to tell just by looking at the wires. (Frequently sold as copper with the only giveaway being "CCA" on the boxes if you're lucky and they haven't managed to get rid of that marking too.)
I was working at a major construction firm and got a call from a job site office trailer that they had lost their internet. The other 2 trailers on the site were still online. I told my boss and he decided to take a drive over there to see what was going on. When he got there he found that the antenna on the office had fallen off the post due to rusted out supports and no one had noticed. He managed to get it back up to the point where they could receive a signal and later had it repaired properly.
Hamish Carmichael published two books in the 1990s containing stories from people who had worked for the many companies that had merged to form ICL. They used to be on the web too.
One story related to a punch card equipment factory in Letchworth during WW2.
One day the air raid alarm went off. The volunteer gun crew downed tools and rushed up to the roof with their machine gun. Just in time to see a low flying bomber disappearing into the distance.
In the 1980s I worked in Site Services department for a large electrical manufacturer, there were three different divisions on the site, two on one side of a public footpath, and my employers on the other. It was decided at very high level that we would install an energy management system to control the lighting, heating, etc, and to read and report energy usage. It was further decided that the three divisions should deal together with one contractor to save costs. I specified that the outstations should be individually connected by fly leads to sockets on a central spine in each of our buildings, and the spines should be run in steel conduit to prevent accidental damage in the factory environment. The other two divisions' Site Services departments thought this too expensive, and simply daisy chained the outstations together using multicore cable strung along the structural steelwork. Once the systems were up and running, we had no trouble, but the others were constantly plagued by disconnections caused by mechanical damage to the cables, rodent activity, and even, in one case, the cable being burnt by a shower of welding sparks from roofing contractors on the outside of the building. Needless to say, a complete reinstall was decided upon, using the same method that I had originally used.
I worked for a large ISP whose data centre was about 300 yards from the NOC office, and I was one of the 24 hour team. The data centre had a UPS, which was supposed to telephone my team with a voice message "Clarkes Avenue UPS failure" (Clarkes Avenue is a pseudonym), if a potential show stopper fault developed. More often than not, the message would be four quiet beeps of successively lower frequency instead of the voice message.
Anyway, the batteries developed a serious fault causing a hydrogen sulfide build-up in the data centre office. The UPS was decommissioned, replaced with a new one in the data centre itself, consisting of a multitude of individual UPS stacks. It had not been fully commissioned.
One Sunday afternoon, during my shift, the alarm company called to say "The alarm system is reporting fire, this is a landlord response." - so off I went to Clarkes Avenue. On arrival, I found that smoke escaped from the door if I opened the one into the data centre. I closed the door and called the fire brigade, also reporting to my shift partner. He raised the issue with the duty management team.
The fire brigade ventilated the building, There was no actual fire but the control box for "blue 2" UPS had had its plastic case burned off. That UPS was in bypass, as it happened. I had to stay at Clarkes Avenue for most of my shift as the alarm was not resetting - the engineer eventually came out and unwired the fire circuit, then reconnected it, and all was well.
Another, different UPS was installed at Clarkes Avenue a few weeks later.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
