back to article Thieves take out Cable & Wireless centre

Cable and Wireless's Watford network site went offline at 1am this morning, after thieves apparently stole vital equipment. Several Reg readers emailed us to report the outage when websites hosted at the Watford centre suddenly went kaput. They were told robbery or vandalism caused the downtime. The outage has been blamed for …

COMMENTS

This topic is closed for new posts.
  1. Colin Wilson

    Backs onto a holiday site?

    In Watford??? Surely some mistake

  2. Tony Dunlop
    Thumb Down

    ft.com is also crimped

    ft.com was affected by this break-in also, and is currently running off servers in the US showing yesterday's content. Only the Alphaville blog at http://ftalphaville.ft.com/ is active on ft.com thus far today.

  3. Anonymous Coward
    Anonymous Coward

    Hmmm.

    Watford = Hertfordshire! so " said one Bucks resident " eh? What does it have to do with the people of Milton Keynes?

    "which locals say backs onto a holiday site" = One of those law abiding, cheeky, Traveller Site type of Holiday Sites?

  4. Anonymous Coward
    Coat

    Single point of failure

    It's a building approx 10miles away from the data centre that was where the break in happened. This is where all the bandwidth for the datacentre arrives.

    Insecure single point of failure and an unacceptable one at that. Irrespective of that fact that this was theft. Diverse paths are required into a facility like this. Just like power.

    C&W limps from crisis to crisis, recently their operation in Leeds was taken down by connectivity failure. The Air conditioning in Watford failed shutting down the hosting rooms and taking many customers off-line a few months back. And Watford suffers from Internet outages from time to time, only after risks have been taken with single points of failure are exposed.

    They are a disgrace.

    Why have I posted anonymously.... I'm a customer in this facility

    My coat is the one with the servers , racks and comms equipment looking for a new home.

  5. Anonymous Coward
    Anonymous Coward

    @ Colin

    Holiday in the 'travelling' sense.

  6. Craig

    Holiday site?

    Is that some kind of euphemism for travellers? (or somebody misunderstood "caravan site"!)

  7. Jamie
    Linux

    Cable thesft in Canada.

    Was reading on a Canadian news site last week about a cable theft. Problem was the idiots tried to steal live electrical cables and became a bit frazzled in the process. Love the irony and no I don't feel bad for someone stealing causing thousand inconvenience because they are too lazy to get a real job.

  8. Joe K
    Alert

    Bit daft

    Where would you even sell this stuff? Ebay is too obvious, you'd get nicked quick enough.

    And if you sit on it for a few months till the heats died down, the things soon turn obsolete in the fast moving tech world.

    They'll nick anything, fuckers.

  9. alex cee

    @Jamie our substation earthing cables got nicked as well!

    On top of this, Our substation earthing cables got nicked a few weeks back, and we where without power from about 5PM to 2AM!!

    'EDF representative, Tracey Sparling, said:"Unfortunately EDF Energy had to isolate supplies to several thousand customers in Watford at 6.22pm yesterday because vital earthing equipment was stolen from four 132,000 volt grid transformers."'

    http://www.watfordobserver.co.uk/news/2314309.thieves_cut_power_to_thousands/

  10. This post has been deleted by its author

  11. Barry

    Not me!

    I live/work about 15min walk from there!

    It wasn't me, I swear!!

    I'll keep my eye out for white tracksuit+cap wearing hoodies trying to offer me Cisco kit 'round the town centre.

  12. Sarah Bee (Written by Reg staff)

    Google maps

    Folks, if you're gonna post a link to a Google map, please run it through Tiny URL first or it ballses things up a little over here. Ta.

  13. Anonymous Coward
    Coat

    Cableless & Wireless

    That is all.

  14. Jim Butler
    Alert

    Not just electrical cables

    A few weeks ago some thieves tried to steal a great big pipe from the cellar of the hospital i work at. As they sawed away they heard some hissing and gave up. The nice big pipe was the gas main feeding the entire hospital.

    The gas leak was discovered by two people sneaking into the cellar for an illicit fag (banned on hospital property anyway). Fortunately they didn't try to light up!

  15. amlendu
    Stop

    Cable & Wireless =

    Cable and Clueless

    Stop before using C&W

  16. Steve
    Unhappy

    @ Sarah

    But we won't be able to see where the link is pointing if it's done via Tiny URL!

    Could be a trick!

  17. Anonymous Coward
    Dead Vulture

    A little unfair

    Don't some of you think it's a little unfair to blame C&W for getting robbed? What's next "It was his own fault he got mugged. Walking down the street - he was asking for it"?

  18. Sarah Bee (Written by Reg staff)

    Re: Google Map URLs

    Yes, I very personally dissed that URL and made it cry. Of course a simple return-click will do just as well.

  19. Anonymous Coward
    Anonymous Coward

    @Steve

    http://tinyurl.com/preview.php

  20. JB
    Stop

    Scrap Metal

    Yes, these idiots will steal anything rather than get a proper job. We recently hired a skip and filled it with household crap. The next morning we woke up to find two blokes clambering about on the skip with about 5 kids hanging around filling up a supermarket trolley with any metal bits they could find. We thought "not doing any harm' and left them to it. Three hours later they were still there. We rang my brothers; you should have seen the chavs' faces when a car pulled up and three huge blokes climbed out! They couldn't get that trolley down the road quick enough. Priceless! Now you see "No Scrap Metal' or "Building Waste Only" signs on all the skips round here. Sorry, a bit off topic, but there you go.

  21. Anonymous Coward
    Paris Hilton

    @ A little unfair

    Nobody is blaming cable & Clueless for getting robbed.

    However they are blaming them for having ALL their connectivity running through a single point, thereby creating a single point of failure, which is a absolute cardinal sin which cannot be forgiven.

    Hell when I worked for a (largeish) local newspaper we even had comms links from different suppliers running from opposite ends of the building, to preclude a careless JCB driver (or slightly less careless carbomber) from severing our connectivity

    It's in the "Little Book of Datacentre Design" right at the very start, beside things you will require:

    1: A secure building (usually involves security staff and cameras)

    2: Diverse power supply

    3: Diverse comms links

    4: Trained staff

    5: Comms Equipment

    6: Servers

    7: Customers

    Paris, cos even she could do better

  22. Anonymous Coward
    Thumb Down

    C&W

    Stolen equipment has been replaced by C&W engineers - my service has been fully restored and I suspect the same goes for anyone else affected by this outage.

    One hopes there is increased security from now on - I'm sure it is standard practice to not rely on one data centre.

  23. Stewart Haywood

    Fewer customers

    Take a look at this page on the C&W website.

    http://portal.cw.com/wps/portal/internet

    For the lazy I quote "OUR FOCUS Is on fewer customers....."

    If they keep up this performance, it looks as though it may be a realistic focus.

  24. kain preacher

    Some thing to think about

    For all of you that says this single point of a failure should of not happened think about this. It was the cable that was stolen. In certain areas in my state only one company owns the physical copper cables. So it does not matter which company you use, it all goes over one companies copper. That company has a problem nothing you can do. No one else has copper.

  25. pctechxp

    Hosted at the site?

    If it wasn't a data centre how can anything be hosted there unless you meant that it hosts connectivity for servers powering several major websites?

    The servers are obviously elsewhere unless the DC is right next door?

    Correct me if I'm wrong but I dont remember comms gear ever being able to host a website apart from its own management interface perhaps?

  26. Mark Bailey
    Pirate

    Why not wait to get the whole story?

    No matter how many diverse connections power feeds you have don't they have to converge at some point to provide service to a single point?

    To those that are smug about their diverse feeds they may leave your premises with 180 degrees of separation but I bet you at some point they run side by side even though your provider may not want to admit it ;)

    I am not saying if C and W are to blame or not, just it is easy to point fingers and by the looks of the previous comments it looks like the band wagon is already overloaded with the righteous :p

  27. Anonymous Coward
    Pirate

    Breaking news - they've lost it again

    The thieves will know C&W needs to get everything up and running again today. Over the weekend they will come back in and take the new kit.

  28. Anonymous Coward
    Anonymous Coward

    MCI

    sted Thursday 10th July 2008 22:12 GMT

    Pirate

    No matter how many diverse connections power feeds you have don't they have to converge at some point to provide service to a single point?

    To those that are smug about their diverse feeds they may leave your premises with 180 degrees of separation but I bet you at some point they run side by side even though your provider may not want to admit it ;)

    I am not saying if C and W are to blame or not, just it is easy to point fingers and by the looks of the previous comments it looks like the band wagon is already overloaded with the righteous :p

    When I worked at MCI hosting center, they had lines coming in from sprint ATT as well as their on lines. They all came into the same side of the building. You better believe that at some point, out side the building those lines ran side by side. This was more than just an ordinary data center. This is were MCI networks had a physical hand off and ties in to ATT lines as Teir 1 provider . If you were to cut the lines there, you would cut the wires here you would screw up routing between ATT and MCI in the area. If you are missing my point here it is. If you cut the lines in the right place you will cause a single point of failure. All providers have to tie into some else at some point. Break that link and see what happens. Oops ATT cant acces any one on MCI. , oops ATT cant access level 3

  29. tony trolle

    re:diverse connections

    I knew of an major companies datacomms center (one of about 12) had a single point of failure sitting in a greybox on a small patch of grass which was in the middle of a busy traffic intersection.

    Story goes if was only found out after a manager had a little chat with one of 'pans people' over morning tea break (guess the company).

    The fix, in only the way it can, ran a new fibre over 50 miles the other way sharing the costs with the exchanges on the way.

  30. Frank

    @Mark re. whole story

    "....but I bet you at some point they run side by side even though your provider may not want to admit it ..."

    Oh yes!. That reminds me of a cautionary tale..............

    A large facility I used to work for had a requirement to carry critical data from one of its existing sites to a new site about 150 miles away. They cleverly specified 180 degree separation at the new site and also contracted two separate suppliers to make the links. To ensure that the suppliers did not use any common carriers, they had to submit a detailed list of all their subcontractors for checking and approval by the client- etc. etc.

    Only when it was all installed and working did someone find out that both suppliers had independently decided to run their cable across a particular road bridge, in conduit, along the same side of the bridge. All it would have taken was a simple traffic accident involving a large van, to rip through the conduit and take out both data links.

    Lessons: Involve your critical suppliers in your technical design process. Review and check everything yourself. Trust no one.

  31. Anonymous Coward
    Anonymous Coward

    holiday caravans

    There was a night theft of lead from our local church in the village. The next day it became obvious from the white footprints, that the Holiday Caravaners who lived in a house backing onto the church were the culprits! Some more caravaners were caught out when a wood chipper was found in a wood backing onto their holiday park which had a tracking device fitted. That was not all that was found either!

  32. Danger Mouse
    Coat

    Satellites

    The bestest (sic) DC's I've hosted in have fail-over satellites dish uplinks. Admittedly it costs a bit but there is really no real excuse for lost of connection when there are failover options, you get what you pay for. If i was responsible for large websites like Sainsburys or the FT you can bet your bottom dollar that I would ensure that it was hosted in a DC with satellite failover.

    Mine's the one with the smug c*nt sticker on the back.

  33. Anonymous Coward
    Thumb Down

    C&W woes

    We were one of the sites take out by yesterday's "incident". We get major issues with C&W every 3 months, usually resulting in significant downtime. Day-to-day maintenance is excruciatingly slow and difficult. In short, these guys are muppets.

  34. Anonymous Coward
    Thumb Down

    @everyone wittering about single points of failure

    Diverse feeds / power etc:

    "at some point they run side by side" - not if they're diverse. You have them in different ducts coming from different exchanges / transformer stations.

    "one company owns the physical copper cables" - even so, you get two feeds from that company, in different ducts, coming from different opints on their network.

    Lets get with the programme here....

  35. Anonymous Coward
    Anonymous Coward

    Failovers cost.

    Yes it might be nice to have very expensive satellite based redundant links, but would you be prepared to pay for it?

    Unless you work for a service provider then you will probably be quite surprised at the number of customers who won't even pay for backups never mind a full business continuity plan.

    I work for a big service provider and we have some very big, household names along with a lot of government work on our books and not all of them will even pay to have redundant links into their network,so they are never going to put their hands in their pockets to pay for an expensive solution like satellite links.

    It's easy to post on here what you would do, but if it was coming out of your own pocket I would bet my house that you wouldn't pay for it, unless the cost could be mitigated by the amount of money you would loose should your website be unavailable. Even then you'd probably find something better to spend your money on.

    For the one person who was wondering, yes we were effected by this. I had to laugh as just last month another one of our third parties has their cables stolen causing an outage.

  36. pctechxp

    @Danger mouse - dish links

    Would agree entirely but even if a provider did not want to go to the expense of providing an uplink to Inmarsat or one of the other providers (inmarsat was one that came immediately to mind but I know there's countless others) there are now more cost effective terrestrial options such as HSDPA or SDSL which can be used (though of course it would be much slower) though Sat is undoubtedly the best option for connection security.

    As regards power issues, the only real way to secure power for a facility is to have onsite generators (where I work we supposedly have 3 diverse feeds from various parts of the town along with 3 substations onsite but we still have trouble if there's an issue with the grid locally and have had to manually switch to generator power on occasion.

  37. Red Bren
    Coat

    @AC

    "Stolen equipment has been replaced by C&W engineers"

    Like that episode of Dr Who where the clockwork repair droids used the crew's bodies to fix the ship?

    <- Mine's the one with the long scarf

  38. Egbert Nobacon
    Flame

    poor internal communications

    The biggest problem for the customers hit by this, was the useless account management and information customers got from C and W frontline.

    We were told by our account management that Watford had been repaired, yet the helpdesk people told us that we were re-routed via a failover to another C and W office!

    C and W have a good network, and good network engineers/people on the ground. Unfortunately account management and the admin and front office are f useless.

    Our account manager asked us what he wanted us to do when we called.

    Er. How about updating us every hour?

    How about giving us an estimated fix time? We kept being told "don't know", all we wanted to know was: Matter of hours? Matter of days? Matter of weeks?

    So we could respond accordingly and start configuring our own alternative routes for traffic.

    C and W are let down by their management, not their network, or their engineers who are doing a fine job. I hope Sainsbury's maul them.

  39. Anonymous Coward
    Unhappy

    Cables side by side

    Common practice, it is known as a flattened ring (Cables in same duct) to at least try and fool the customers that it is a protected ring!

    Increasingly common is the practice of getting wavelengths from 2 different providers only to find they share the same physical fibre. Happens ALL the time in EVERY telco so dont be suprised to hear of more "unforgivable" single points of failure

  40. Anonymous Coward
    Thumb Down

    Vandalism = junk food opportunity?

    Cowboy and Witless supply the ISDN to the UKs Post Offices (that aren't yet or can't be on ADSL) meaning hundreds of sites were down... meaning I had to escape the PO's Helpdesk and go and eat doughnuts in the hub cupbaord for a couple of hours whilst phones exploded.

    Bringing it back to a relevant angle, you have here second hand experience that C+W said what the problem was quite quickly: "naughty vandals" and "engineers ready to replace equipment as soon as the police piss off" was the theme, but they steadfastedly refused to give us any eta on a fix time. At all.

    Not very good when they're supplying a vital service for a British Institution like the PO.

  41. Anonymous Coward
    Alert

    Exactly when ...

    did the greater London area degenerate into a third world nation?

    All these stories of stolen cables, scrap salvaging, dumpster diving, its like its still Dicken's London over there. You think the USA is so bad but maybe if you shot a few of these criminals you'd have less problems.

  42. richard tanswell
    Stop

    Backs onto a holiday site

    I live in Watford, I didn't realise I could holiday there! Wow what a carbon footprint saving I can make!!

    I think what they actually mean is "traveller's site" but in a PC way! Now that the "travellers" have damaged their local internet access, how are they going to do their Ebay listings?

  43. Anonymous Coward
    Flame

    Read between the lines, FFS...

    ... don't you understand what they are talking about??

    "Holiday site" is a very unusual term... and why would a "holiday site" be "raided by police only last week, looking for scrap metal theft evidence"?

    Think why the word "holiday" might be relevant... think "caravans"...

    Now think of "A site in Watford on which there are caravans, where it would not be a massive surprise to find evidence of theft of scrap metal, which attracted a large posse of Her Majesty's finest - with dogs and wearing personal protective equipment (PPE) in order so to do..."

    Geddit?

    God, PC-speak is destroying the meaning of plain English, simply in order to ensure that minority copper thieves... ermmm... I mean, people of a travelling persuasion... don't have their sensitive egos offended against... :-(

  44. pctechxp

    pice of copper + greed = crime

    with copper and other metals at a high price there is bound to be instances like this (which of course doesn't make it right)

    I blame the speculators in London and New York for pushing up the price of metal such as copper to a ridiculous price.

    But pity the poor FT reader who cant get his fix, maybe he needs to find something more worthwhile to read that doesn't chargw a subscription (you are reading an excellent example of such a publication right now)

  45. This post has been deleted by its author

This topic is closed for new posts.

Other stories you might like