back to article Flood, fire at BT Paddington node causes widespread problems

A fire at a BT building in central London is causing widespread landline, internet and mobile network problems, according to reports. A blaze at Burne House in North Paddington was reported this morning. According to Gradwell, a business ISP, 437 local exchanges and up to 37,500 Datastream circuits have been affected. It said …

COMMENTS

This topic is closed for new posts.

Page:

Unhappy

Birmingham down as well

Had problems with DSL in Birmingham from midnight, BT are trying to resolve but have had to divert resources to Paddington to sort the problem there. This from my ISP Entanet

http://noc.enta.net/

0
0
Unhappy

the day the phones died

I was gonna write in as nothing was reported for a while - this is majorly affecting us - cant make any outgoing calls and the mobile phone network is on it's knees at the moment.

0
0
Flame

Easynet are affected too

Easynet are reporting their Paddington, North Paddington and Marylebone exchanges down due to 'flooding'. A little overexuberance on the part of the fire brigade?

0
0
Grenade

No, A little overexuberance on the part of H&S act

Two factors at work here.

1. Nobody wants to risk halon, co2 or other methods of fire suppression which take out the fire by smothering it. The UK H&S act pretty much took care of that. If someone dies, you are guilty by default regardless of how much measures did you put in to prevent this. This is rather silly as the person can die in fire or from electrical shock once the water goes in same way as from Halon or Co2. In fact if there are cabinets with masks at every X meters in any direction Halon or CO2 are inherently safer than water. That however is impossible to explain to the idiots from Connect and other union w*nkers. So end of the day you put sprinklers in knowing that any equipment on that floor unaffected by fire will be destroyed by water. This happens on casual basis in all providers - BT has had it before, IIRC Level3 has had it, either Globix or Global Lossing had it as well.

2. The power distribution architecture in nearly all colocation facilities in the UK does not have per-rack cut-off. You do not really need to turn on sprinklers if you have per rack cut off for electricity and cooling so you can stop electricity as well as pumping fresh air to the rack in question and X neighbouring racks for 5-10 minutes before you push the big red button. It is not that difficult to plan, build and even retrofit a datacenter so you can take down completely sections from it. Considering that a rack of telecoms equipment can easily cost north of half a million pounds the common ineptitude in power and cooling design by UK telecoms operators is outright criminal. Though once again some of that can be blamed on H&S and the way it has been exploited by unions under Nu LieBore.

0
0
FAIL

re: No, A little overexuberance on the part of H&S act

Actually, many data centre facilities use a high fog mist system to suppress fire which is entirely different from "sprinklers". It's also a hell of a lot safer than halon or co2 based suppression methods.

Methinks you've never actually worked in a data centre from the nonsense you've just spouted, I in fact have.

0
0
Bronze badge
Flame

internet redundancy

I thought that the interwebs were supposed to be robust enough to suffer nuclear destruction of some of it's connection nodes....

If one fire in one exchange was enough to bring down the national communication infrastructure, maybe the USSR should have invested in napalm rather than nukes...

2
2

Well you're still talking, no?

Part of the network is bollocksed and you're still posting...

Or are you one of those Londoners who thinks the world ends at the M25, and the "national communication infrastructure" is 20 miles across?

No problems here in Leeds.

3
0
FAIL

Nope.

"I thought that the interwebs were supposed to be robust enough to suffer nuclear destruction of some of it's connection nodes..."

Then you thought wrong. HTH

0
0

Inderect Effects

Really.

Try using your credit card online then.

S.

0
0
Silver badge
Boffin

re: internet redundancy

As has been pointed out, you'll find the interwebs are running just fine thanks. If you put an axe through your router at home, you'll find much the same effect.

The idea of the nuclear defended network is so that the rest of the world (well, US) could still talk even if, say, Washington DC was suddenly replaced by a crater. Bear in mind also that the Internet is not a singular entity. A better description is a network of networks. One of the subnetworks is out - that's all.

If you think this is bad, consider what would happen if Telehouse Docklands were to disappear. Fortunately the more important a node, the more redunancy built in.

2
0

Re: Well you're still talking, no?

"Or are you one of those Londoners who thinks the world ends at the M25, and the "national communication infrastructure" is 20 miles across?"

I'm shocked, shocked I tell you. Always thought it ended at the north circular.

0
0

you are badly misinformed

Your theory is far from practice. Replace london with a crater and then come tell me if the rest of the UK will be able to access anything. The chances are, all the peripheral nodes will go down as well just from the lack of routing paths. Yes, you will find that many times your packets take some pretty long routes in order to reach their destination and those routes usually go through some main exchange points.

0
0
Happy

That far?

If it's outside the Circle Line it's foreign.

0
0
Flame

"No problems here in Leeds"

It ain't the Paddington node that's burning - it's your pants!

; P

0
0
Silver badge

Almost...

You're confusing "end of the world" with "edge of civilisation".

0
0
Silver badge

Open-loop and leaky

Have you not seen that the Circle line is no longer just a simple loop?

http://www.tfl.gov.uk/corporate/media/newscentre/11300.aspx

0
0
Alert

Any word on casualties?

Hoping for good news, did the BT staff get out safely?

2
0
Silver badge
FAIL

It probably tells us quite a lot about BT's infrastructure

If data for other parts of the country are being routed via central London, but they can't be more specific about whom or where. I'm just glad I'm no longer a customer of theirs, for so many reasons. This is despite the monthly 'Come Back to BT!' junk mail I seem to receive. Sometimes I don't know how I summon the willpower to overcome its hypnotic effect...

1
1
FAIL

Redundancy

Lucky they have redundancy built-in in case of things like this. What's that? People all over the country affected? Oh....

2
1
Stop

@Bilgepipe

"What's that? People all over the country affected? Oh...." No they are not, thats WHY we can ALL talk aout this online...

1
0

do'h

I wondered why I had no 3G signal on my phone (vodafone).

0
0
FAIL

More details, incl. area codes

http://status.zensupport.co.uk/index.php?serviceid=5&incidentid=1729&simple=1

0
0
Unhappy

Do they never learn?

There was a fire in the Scarborough exchange in the 1990's, caused complete havoc for everybody. Turned out to be the sprinkler system was fault.

Wonder what this one will be?

0
0
Flame

why would anybody...

... put anything fire sensitive in Burne House?

This was inevitable, if you ask me.

2
0
FAIL

Nice Redundancy BT!

Wow, I know this is a major exchange but surely this is the reason this kind of thing is planned for. I wouldn't mind but we were running a DR scenario today that has been pulled due to this, irony indeed.

We've lost one satellite office and it is still chunking up our network nicely - If anyone is interested BT have informed us that they should have a solution in place by 16:15 - Although by the sounds of this article I am beginning to doubt it.

0
1
Gold badge
FAIL

Not again

How long since that fire in Manchester took out most of the traffic to the US?

What is it with BT? Arsonists, smoking *inside* the building, soldering with blow torches?

No flame but really not a good show at all.

1
0
Flame

Paying

SagePay not working this morning. Their website says problems due to external provider. putting 2 and 2 together...

0
0
Flame

No landlines or broadband in Hyde Park

Yup it's true - Hyde Park is devoid of landlines or broadband and business has drawn to a standstill. Mates in West London offline too!

0
0
Anonymous Coward

Re: No landlines or broadband in Hyde Park

So it's like a park then?

2
0
Silver badge

Hmm

That's one fuck of a big exchange, lots of major routing through there - sort of line of sight up the Edgeware Road towards Birmingham.

One would assume that there are traffic jams as the rest of the system tries to re-route it all - think of the M25 being suddenly closed around Dartford Crossing on a bank holiday.

Not sure that redundancy would help in this case as it sounds like it's not just one or two major links that have gone.

( there are times when I almost wish I was back there - a few key presses and clicks and I'd know what was out)

0
0
Joke

Re: Redundancy

"Lucky they have reduncancy built-in"

Of course they have redundancy .... or has BT stopped handing out P45s ?

3
0
Flame

SAGE PAY Down Again!

http://www.sagepay.com/system_monitor.asp

Seems that this is affecting Llyods cardnet customers via Sagepay

Would never have happened under protx! (thats a Joke)

0
0
Flame

SagePay

This, according to SagePay website:

http://www.sagepay.com/system_monitor.asp

31 March 2010 12:02:45 - Please be advised Lloyds Cardnet and Halifax Bank of Scotland customers are currently unable to process transactions via the Sage Pay gateway.

This is due to a BT communications network failure, and not an issue on the processing platforms.

We are currently awaiting an update from the respective parties.

0
0

This post has been deleted by a moderator

Go

PayPoint Still Works thou

SagePay might be broken, but PayPoint.net is still working, albeit at a slightly reduced capacity. Talking with support it seems their backup connections are taking the load pretty well.

0
0

More Inderect Effects

Expect other card services to be impacted.

This from a SagePay communication email:

Hello,

We'd like to update you on an external issue which is effecting nationwide telecomms/datacomms.

British Telecom has confirmed a fire at their Exchange in the Paddington area of London. This has impacted a huge number of UK businesses, including Sage Pay and the major banks that route transactions through the Exchange. Sage Pay has taken action on your behalf by quickly moving transaction processing to our backup data centre. However, the issue has affected connectivity to some banks who rely on the exchange and we are currently unable to process via those banks at this time.

Please note, as banks and payment providers look for alternative transaction routing solutions away from the Paddington Exchange, this could create some general capacity issues, resulting in transaction 'time-outs'.

0
0
Headmaster

Candle Power

Somebody tell Municipal Lighting we're going to candle power in ten minutes! Where's Richard Dreyfuss when you need him?

1
0
Thumb Down

Sagepay 0 - 1 Worldpay

Sagepay are a pile of p***!

Thank goodness we have not moved completely over to Sagepay! Worldpay saves the day, again.

0
0
Unhappy

Colt netwokrs...

Some of our Colt links are down as well...

0
0
Thumb Up

My internet has never been faster

Wow massive speed increase!

0
0
Flame

O2 as well

O2 have 117 cell sites down across london due to this, if anyone's interested!

0
0

Sage Pay Still have problems with HBOS * Cardnet

Incident Report from SagePay:

At 9:50am this morning we had problems connecting to the authorising banks.

We investigated and subsequently moved our authorisations to our DR site which was processing at 9:59

After a brief period of processing the authorisations slowed, this was a result of many people moving to backup configurations to route round an issue that transpired to be a fire at a main telecoms/datacoms exchange in docklands.

Within minutes the bottle neck had eased and we were processing transactions for all banks except HBOS and Lloyds TSB. We switched the destination NUA’s to the backups and there was still nothing being processed by those banks

We made the appropriate calls and even though our backup X25 system worked perfectly it transpires that the bureau responsible for processing HBOS and Lloyds transactions did not have resilience outside of that exchange.

We have been working with the bureau and alternative to X25 in order to resolve the issue.

As of 15:15 we now have a workaround in place, however with the X25 network working at a much reduced capacity we are experiencing delays in processing transactions

0
0
Pint

Hmmmm

Must have been the un-relenting, raw fury of countless IT people manifesting itself as the FLAMES OF HELL THEMSELVES after having their internet connections hijacked to force BT marketing tripe down our necks.

Or something else, probably.

0
0
Thumb Down

Ugh Sagepay

$work uses Sagepay for taking payments on our retail website, which has seen a drop in orders because of this issue and it also doesn't help that we also use Zen for our ADSL which is experiencing very slooooow downloads :(

0
0

Oh the horror!

Can someone please reassure me that they managed to save the Marmalade Sandwiches from destruction!

0
0
Anonymous Coward

Redundancy Costs

Every time there is an outage of this sort there is at least one commentard who goes on about redundancy. Last week we had an outage at one of our sites which took down chip and pin for almost a whole day, this was down to an equipment failure at a BT exchange which, to be fair to BT they worked into the early hours to fix. However we got it in the ear from our customer because they said that (a) they should have been warned in advance so they could implement their BCP and (b) the system should be redundant to prevent this sort of thing happening.

We answered (a) that the point of your BCP is that it's a robust plan to account for emergencies and you don't often get warning of emergencies, unless they happen to be caused by an earthquake and you happen to speak toad. And (b) that BT could probably make every bit of kit and every single circuit redundant, they could have redundant power supplies and diverse routed circuits and it would all cost a fortune.

If BT or your ISP increased your bill by 100% or more to implement this redundancy would you be happy to pay?

The internet is packed full of redundancy. To use some old terminology if a hub site goes down somewhere the rest of the network will route around it, but you still have a big hole around that site. Any edge site that connects into that hub will be down until the hub site is back on line. If the end user wants redundancy then they need to connect to more than one hub site.

3
0
Thumb Up

Why didn't you have redundancy?

Well said.

Typically customers look at the cost of a redundant/diverse network and decide it's too expensive. Then they get an outage like this and re-evaluate the business risk. Go figure.

Biggest SPF on networks are the single circuits to customer and for retailers or small businesses running on xDSL, that's very difficult to provide any kind of diverse path back to the exchange. But it's cheap, until it goes wrong and you can't trade. Linked to that is the xDSL termination at the ISP side. Those circuits are expensive, so if ISP's don't have diverse connectivity to multiple terminating sites, there's another SPF but that one is more outside of the end customer's control, unless they ask their provider the right questions.

Unfortunately it often takes outages like this for customers to reconsider their network design, BCP and whether the cost of a resilient network solution is right for them. YGWYPF.

0
0
Joke

no problems...

Passing this story to a good friend, he politely informed me that BT must have got the news wrong....apparently there are no problems on the Internet: http://status-cable.virginmedia.com/vmstatus/summary.do

0
0
FAIL

good report on the bbc

oops.

http://www.ninja-monkey.co.uk/byte/bbcfu.jpg

0
0

No cards on TfL or at McDonalds either

DR managers might hear the sound of P45s dropping on their desks at TfL who cannot accept any cards anywhere on their network and McDonalds who have similar problems in London.

0
0

Page:

This topic is closed for new posts.

Forums