A fire at a BT building in central London is causing widespread landline, internet and mobile network problems, according to reports. A blaze at Burne House in North Paddington was reported this morning. According to Gradwell, a business ISP, 437 local exchanges and up to 37,500 Datastream circuits have been affected. It said …
Birmingham down as well
Had problems with DSL in Birmingham from midnight, BT are trying to resolve but have had to divert resources to Paddington to sort the problem there. This from my ISP Entanet
the day the phones died
I was gonna write in as nothing was reported for a while - this is majorly affecting us - cant make any outgoing calls and the mobile phone network is on it's knees at the moment.
Easynet are affected too
Easynet are reporting their Paddington, North Paddington and Marylebone exchanges down due to 'flooding'. A little overexuberance on the part of the fire brigade?
No, A little overexuberance on the part of H&S act
Two factors at work here.
1. Nobody wants to risk halon, co2 or other methods of fire suppression which take out the fire by smothering it. The UK H&S act pretty much took care of that. If someone dies, you are guilty by default regardless of how much measures did you put in to prevent this. This is rather silly as the person can die in fire or from electrical shock once the water goes in same way as from Halon or Co2. In fact if there are cabinets with masks at every X meters in any direction Halon or CO2 are inherently safer than water. That however is impossible to explain to the idiots from Connect and other union w*nkers. So end of the day you put sprinklers in knowing that any equipment on that floor unaffected by fire will be destroyed by water. This happens on casual basis in all providers - BT has had it before, IIRC Level3 has had it, either Globix or Global Lossing had it as well.
2. The power distribution architecture in nearly all colocation facilities in the UK does not have per-rack cut-off. You do not really need to turn on sprinklers if you have per rack cut off for electricity and cooling so you can stop electricity as well as pumping fresh air to the rack in question and X neighbouring racks for 5-10 minutes before you push the big red button. It is not that difficult to plan, build and even retrofit a datacenter so you can take down completely sections from it. Considering that a rack of telecoms equipment can easily cost north of half a million pounds the common ineptitude in power and cooling design by UK telecoms operators is outright criminal. Though once again some of that can be blamed on H&S and the way it has been exploited by unions under Nu LieBore.
re: No, A little overexuberance on the part of H&S act
Actually, many data centre facilities use a high fog mist system to suppress fire which is entirely different from "sprinklers". It's also a hell of a lot safer than halon or co2 based suppression methods.
Methinks you've never actually worked in a data centre from the nonsense you've just spouted, I in fact have.
I thought that the interwebs were supposed to be robust enough to suffer nuclear destruction of some of it's connection nodes....
If one fire in one exchange was enough to bring down the national communication infrastructure, maybe the USSR should have invested in napalm rather than nukes...
Well you're still talking, no?
Part of the network is bollocksed and you're still posting...
Or are you one of those Londoners who thinks the world ends at the M25, and the "national communication infrastructure" is 20 miles across?
No problems here in Leeds.
"I thought that the interwebs were supposed to be robust enough to suffer nuclear destruction of some of it's connection nodes..."
Then you thought wrong. HTH
Try using your credit card online then.
re: internet redundancy
As has been pointed out, you'll find the interwebs are running just fine thanks. If you put an axe through your router at home, you'll find much the same effect.
The idea of the nuclear defended network is so that the rest of the world (well, US) could still talk even if, say, Washington DC was suddenly replaced by a crater. Bear in mind also that the Internet is not a singular entity. A better description is a network of networks. One of the subnetworks is out - that's all.
If you think this is bad, consider what would happen if Telehouse Docklands were to disappear. Fortunately the more important a node, the more redunancy built in.
Re: Well you're still talking, no?
"Or are you one of those Londoners who thinks the world ends at the M25, and the "national communication infrastructure" is 20 miles across?"
I'm shocked, shocked I tell you. Always thought it ended at the north circular.
you are badly misinformed
Your theory is far from practice. Replace london with a crater and then come tell me if the rest of the UK will be able to access anything. The chances are, all the peripheral nodes will go down as well just from the lack of routing paths. Yes, you will find that many times your packets take some pretty long routes in order to reach their destination and those routes usually go through some main exchange points.
If it's outside the Circle Line it's foreign.
"No problems here in Leeds"
It ain't the Paddington node that's burning - it's your pants!
You're confusing "end of the world" with "edge of civilisation".
Open-loop and leaky
Have you not seen that the Circle line is no longer just a simple loop?
Any word on casualties?
Hoping for good news, did the BT staff get out safely?
It probably tells us quite a lot about BT's infrastructure
If data for other parts of the country are being routed via central London, but they can't be more specific about whom or where. I'm just glad I'm no longer a customer of theirs, for so many reasons. This is despite the monthly 'Come Back to BT!' junk mail I seem to receive. Sometimes I don't know how I summon the willpower to overcome its hypnotic effect...
Lucky they have redundancy built-in in case of things like this. What's that? People all over the country affected? Oh....
"What's that? People all over the country affected? Oh...." No they are not, thats WHY we can ALL talk aout this online...
I wondered why I had no 3G signal on my phone (vodafone).
More details, incl. area codes
Do they never learn?
There was a fire in the Scarborough exchange in the 1990's, caused complete havoc for everybody. Turned out to be the sprinkler system was fault.
Wonder what this one will be?
why would anybody...
... put anything fire sensitive in Burne House?
This was inevitable, if you ask me.
Nice Redundancy BT!
Wow, I know this is a major exchange but surely this is the reason this kind of thing is planned for. I wouldn't mind but we were running a DR scenario today that has been pulled due to this, irony indeed.
We've lost one satellite office and it is still chunking up our network nicely - If anyone is interested BT have informed us that they should have a solution in place by 16:15 - Although by the sounds of this article I am beginning to doubt it.
How long since that fire in Manchester took out most of the traffic to the US?
What is it with BT? Arsonists, smoking *inside* the building, soldering with blow torches?
No flame but really not a good show at all.
SagePay not working this morning. Their website says problems due to external provider. putting 2 and 2 together...
No landlines or broadband in Hyde Park
Yup it's true - Hyde Park is devoid of landlines or broadband and business has drawn to a standstill. Mates in West London offline too!
Re: No landlines or broadband in Hyde Park
So it's like a park then?
That's one fuck of a big exchange, lots of major routing through there - sort of line of sight up the Edgeware Road towards Birmingham.
One would assume that there are traffic jams as the rest of the system tries to re-route it all - think of the M25 being suddenly closed around Dartford Crossing on a bank holiday.
Not sure that redundancy would help in this case as it sounds like it's not just one or two major links that have gone.
( there are times when I almost wish I was back there - a few key presses and clicks and I'd know what was out)
"Lucky they have reduncancy built-in"
Of course they have redundancy .... or has BT stopped handing out P45s ?
SAGE PAY Down Again!
Seems that this is affecting Llyods cardnet customers via Sagepay
Would never have happened under protx! (thats a Joke)
This, according to SagePay website:
31 March 2010 12:02:45 - Please be advised Lloyds Cardnet and Halifax Bank of Scotland customers are currently unable to process transactions via the Sage Pay gateway.
This is due to a BT communications network failure, and not an issue on the processing platforms.
We are currently awaiting an update from the respective parties.
PayPoint Still Works thou
SagePay might be broken, but PayPoint.net is still working, albeit at a slightly reduced capacity. Talking with support it seems their backup connections are taking the load pretty well.
More Inderect Effects
Expect other card services to be impacted.
This from a SagePay communication email:
We'd like to update you on an external issue which is effecting nationwide telecomms/datacomms.
British Telecom has confirmed a fire at their Exchange in the Paddington area of London. This has impacted a huge number of UK businesses, including Sage Pay and the major banks that route transactions through the Exchange. Sage Pay has taken action on your behalf by quickly moving transaction processing to our backup data centre. However, the issue has affected connectivity to some banks who rely on the exchange and we are currently unable to process via those banks at this time.
Please note, as banks and payment providers look for alternative transaction routing solutions away from the Paddington Exchange, this could create some general capacity issues, resulting in transaction 'time-outs'.
Somebody tell Municipal Lighting we're going to candle power in ten minutes! Where's Richard Dreyfuss when you need him?
Sagepay 0 - 1 Worldpay
Sagepay are a pile of p***!
Thank goodness we have not moved completely over to Sagepay! Worldpay saves the day, again.
Some of our Colt links are down as well...
My internet has never been faster
Wow massive speed increase!
O2 as well
O2 have 117 cell sites down across london due to this, if anyone's interested!
Sage Pay Still have problems with HBOS * Cardnet
Incident Report from SagePay:
At 9:50am this morning we had problems connecting to the authorising banks.
We investigated and subsequently moved our authorisations to our DR site which was processing at 9:59
After a brief period of processing the authorisations slowed, this was a result of many people moving to backup configurations to route round an issue that transpired to be a fire at a main telecoms/datacoms exchange in docklands.
Within minutes the bottle neck had eased and we were processing transactions for all banks except HBOS and Lloyds TSB. We switched the destination NUA’s to the backups and there was still nothing being processed by those banks
We made the appropriate calls and even though our backup X25 system worked perfectly it transpires that the bureau responsible for processing HBOS and Lloyds transactions did not have resilience outside of that exchange.
We have been working with the bureau and alternative to X25 in order to resolve the issue.
As of 15:15 we now have a workaround in place, however with the X25 network working at a much reduced capacity we are experiencing delays in processing transactions
Must have been the un-relenting, raw fury of countless IT people manifesting itself as the FLAMES OF HELL THEMSELVES after having their internet connections hijacked to force BT marketing tripe down our necks.
Or something else, probably.
$work uses Sagepay for taking payments on our retail website, which has seen a drop in orders because of this issue and it also doesn't help that we also use Zen for our ADSL which is experiencing very slooooow downloads :(
Oh the horror!
Can someone please reassure me that they managed to save the Marmalade Sandwiches from destruction!
Every time there is an outage of this sort there is at least one commentard who goes on about redundancy. Last week we had an outage at one of our sites which took down chip and pin for almost a whole day, this was down to an equipment failure at a BT exchange which, to be fair to BT they worked into the early hours to fix. However we got it in the ear from our customer because they said that (a) they should have been warned in advance so they could implement their BCP and (b) the system should be redundant to prevent this sort of thing happening.
We answered (a) that the point of your BCP is that it's a robust plan to account for emergencies and you don't often get warning of emergencies, unless they happen to be caused by an earthquake and you happen to speak toad. And (b) that BT could probably make every bit of kit and every single circuit redundant, they could have redundant power supplies and diverse routed circuits and it would all cost a fortune.
If BT or your ISP increased your bill by 100% or more to implement this redundancy would you be happy to pay?
The internet is packed full of redundancy. To use some old terminology if a hub site goes down somewhere the rest of the network will route around it, but you still have a big hole around that site. Any edge site that connects into that hub will be down until the hub site is back on line. If the end user wants redundancy then they need to connect to more than one hub site.
Why didn't you have redundancy?
Typically customers look at the cost of a redundant/diverse network and decide it's too expensive. Then they get an outage like this and re-evaluate the business risk. Go figure.
Biggest SPF on networks are the single circuits to customer and for retailers or small businesses running on xDSL, that's very difficult to provide any kind of diverse path back to the exchange. But it's cheap, until it goes wrong and you can't trade. Linked to that is the xDSL termination at the ISP side. Those circuits are expensive, so if ISP's don't have diverse connectivity to multiple terminating sites, there's another SPF but that one is more outside of the end customer's control, unless they ask their provider the right questions.
Unfortunately it often takes outages like this for customers to reconsider their network design, BCP and whether the cost of a resilient network solution is right for them. YGWYPF.
Passing this story to a good friend, he politely informed me that BT must have got the news wrong....apparently there are no problems on the Internet: http://status-cable.virginmedia.com/vmstatus/summary.do
good report on the bbc
No cards on TfL or at McDonalds either
DR managers might hear the sound of P45s dropping on their desks at TfL who cannot accept any cards anywhere on their network and McDonalds who have similar problems in London.