back to article The watts in a box that kept West London's lights on

A fire in the Victorian tunnels beneath Kingsway in central West London on Good Friday, in April, pulled the plug on thousands of businesses in a small area. The blaze – which burned for a day and a half – succeeded in damaging 19 high-voltage cables, leading to a significant loss of load on local utility provider UK Power …

  1. Anonymous Coward
    Anonymous Coward

    If this happened anywhere outside London don't expect any help whatsoever.

    1. Bob H

      If you have a DR plan then you should know exactly what support you'll get before this happens. Most companies don't have a DR plan or have an out of date one which doesn't apply to their business any more.

      1. Anonymous Coward
        Anonymous Coward

        My point was that this wasn't DR, UKPN supplied the generators, the power company. Nowhere else in the UK would this happen.

        1. Dave 126 Silver badge

          Population ddenisty.

          I have no opinion, but am curious to explore this. Presumably few UK places outside London could have as many homes and businesses cut off by one single fire. I would also imagine that, for the same reason, few places outside of London would have as many businesses benefit from emergency generators being trucked in.

          When we hear of prolonged power cuts outside of London, it tends to be the result of high winds, floods or other events that disrupt daily life beyond just power, and make the trucking of generators both a lower priority and more logistically challenging.

          1. Anonymous Coward
            Anonymous Coward

            Single point of failure?

            "Presumably few UK places outside London could have as many homes and businesses cut off by one single <incident>"

            Does it specifically have to be fire or is flood allowed?

            In summer 2007 when there was a lot of rain in the Midlands, Walham Switching Centre (which supplies quite a large area) nearly got flooded. The Armed Forces had to be brought in to protect it. A semi-permanent barrier has since been put in place, just in case.

            It was reported at one point that over half a million people would have lost power. Including GCHQ.

            Water supplies were also badly affected (water off for days in places), and (again) the Armed Forces had to be brought in, to distribute bottled drinking water.

            Various reports, e.g.

            http://news.bbc.co.uk/1/hi/england/gloucestershire/6911490.stm

            1. druck Silver badge

              Re: Single point of failure?

              The power was off for up to 4 days, but water wasn't restored for about 3 weeks, not pleasant.

        2. Steve Davies 3 Silver badge

          How do you know?

          As you seem so sure about this why don't you share some proof to those of us who don't know?

          A lot of us work well outside the M25 bubble so would be very interested in finding out more about why UKPN will only do this in London.

          1. jonathanb Silver badge

            Re: How do you know?

            Do UK Power Networks operate outside of London? 40 miles down the road in Reading, Scottish and Southern Energy is our local grid operator.

        3. Phil O'Sophical Silver badge

          this wasn't DR, UKPN supplied the generators

          Where in the article does it say that? MEMS clearly had existing customers in the area who were supplied according to their DR contracts.

          1. Anonymous Coward
            Anonymous Coward

            MEMS power provided the backup generators for UKPN. Some individual companies may have invoked their own DR plans if they couldn't wait for UKPN.

          2. Anonymous Coward
            Go

            I live in a village in the midland and can state they do do this. The difference is you may not notice, as we lost power lines. So they hooked them up after the break and power was restored. However unless you saw the bloody great containers in a field, you would be none the wiser and just presume the lines were fixed.

            BTW, relying on power being shipped in is not a DR plan, but just a bit of one, for one scenario.

            1. the spectacularly refined chap

              I live in a village in the midland and can state they do do this. The difference is you may not notice, as we lost power lines. So they hooked them up after the break and power was restored. However unless you saw the bloody great containers in a field, you would be none the wiser and just presume the lines were fixed.

              Of course they do - they have a legal responsibility to do so, it is just as you note the power density is most locations is much lower and doesn't have the same headline grabbing attention - there is nothing particularly dramatic about the appearance of a trailer in the local church car park. This is on an epic scale but that makes it less likely - the costs skyrocket and compensation clauses kick in much later when a large number of properties are affected.

              My father used to work for a generator services company. They didn't get much of this work since they weren't - they were more sales and service with a sideline in advance hire for events and engineering work than emergency response - but they'd get occasional calls for this kind of stuff across northern England. Even in rural locations - even three or four farmers will play merry hell if they can't milk their cows. The OP is simply making up stuff on the spot - conspiracy theories only work if they are not completely divorced from reality.

        4. Conor Turton

          "Nowhere else in the UK would this happen."

          Yet it has. In Hornsea in East Yorkshire last week the power company stuck a massive generator in a pub carpark when a section of the town lost power. I found out about it when I went to my amateur radio club meeting to be greeted by the sight of this massive generator in the car park with massive cables going into a hole in the street and two workers sat in a van babysitting it.

          1. Dabooka Silver badge
            Thumb Up

            Ah, Horsea

            Lovely part of the world, Ulrome / Skipsea is a frequent destination for our hols. I'd imagine some of the locals haven't seen this new fangled electric lighting before

        5. Anonymous Coward
          Anonymous Coward

          Nowhere else in the UK would this happen

          When I lived in West Raynham (darkest Norfolk) we had a genny sitting below the village power pole being fuelled up by the hire company for about three weeks (within 2 hours of incident) whilst the local supply was being repaired due to a lorry / snow / power line incident on one of the main roads

          1. Nigel 11

            Re: Nowhere else in the UK would this happen

            I think what the uninformed are talking about, is what happens when the weather "nukes" the infrastructure across a wide area, and there aren't sufficient generators and human resources to fix it all within hours. So there's triage. The most economically important bits get fixed first. Which does mean that London will be prioritized over (say) Swindon, which in turn will be prioritised over Lower Cheeseworth (I made that one up).

            1. Intractable Potsherd Silver badge

              Re: Nowhere else in the UK would this happen

              Just to add to the evidence that the OP is talking utter bollocks, here in a small, unregarded village in Fife (seriously, you've never heard of it) wind brought down a tree onto an overhead powerline, taking out several houses' electron supply. Power company workers were there in less than an hour, with a genny in place sometime in the hour after that. It was really impressive how quickly everything just worked.

    2. Anonymous Coward
      Anonymous Coward

      I live in a small village (200 properties). NPG had generators on site and running within a few hours when a major disruption occurred.

      1. Lionel Baden

        when i was a lad

        I used to live in Witley small town near guildford, when the power went out the only way i knew it was on a truck, was because it was parked down our street.

        That being said, Power went out 5 years ago in Horsham, for a period of 6 days, no trucks in sight, they offered us £50~ to replace freezer food.

        Pot Luck I guess

      2. Martin an gof Silver badge

        Small village, small generator

        I live in a small village (200 properties). NPG had generators on site and running within a few hours when a major disruption occurred.

        Our hamlet - 80-ish dwellings, no pub, no shop - is on the end of a long overhead line. We suffer from a fair number of short power cuts, but only on a couple of occasions have they (Western Power) brought in a generator.

        The most memorable of these was a few years back when they were moving and renewing the single pole-mounted transformer that serves the entire hamlet in order to clear a building plot. Warning letters were sent out and at the appointed hour the power went off briefly while they disconnected the old transformer and connected up the generator.

        Everything was fine until tea time. Obviously WPD hadn't realised that since gas only came to the village relatively recently many of the houses still cook on electric cookers and, this being late autumn (IIRC), the ones with electric heating started trying to warm up.

        Said generator started sounding a bit strained, and then tripped out leaving us with a half-cooked pan of chips which we took to the neighbours and finished off on their gas cooker.

        Power stayed off for perhaps three hours after that as the engineers couldn't chance resetting the genny.

        Since then the power has been more reliable, but we still get perhaps three or four short power cuts a year, rarely more than a few minutes, so as well as the small UPS I've long had installed on the computers (note: stepped sine-wave UPSes will not run the pump in a central heating boiler!) I now have a chunkier pure-sine UPS running among other things the printer, which really doesn't like power cuts.

        To get back to the OP's point - that support in London is likely to be more forthcoming - I think the answer is "it depends". I would imagine that if there were an outage in the nearby town (population 30,000+) that also took us out, in the case of a lack of generators they would get priority and we'd be breaking out the candles and torches. If there were a problem affecting half of Cardiff as well, then those 30,000 residents would also be searching for the matches.

        M.

    3. Anonymous Coward
      Anonymous Coward

      Why all this hatred?

      Seriously, is it envy? London is the third biggest city in Europe, so you can expect it to have a very high concentration of people. Any disaster (I'm not sure if I'd call a power cut a disaster, unless you operate a hospital, but anyway) of course affects a much higher number of people than it would in the country side. Naturally any kind of emergency service would have a higher density there too.

      On the other hand, you have better view, nature which isn't confined to crammed parks, better air, and a hell of a lot cheaper cost of living (and a higher standard of living) if you do not live in London. So be happy about that and let it go.

      1. the spectacularly refined chap

        Re: Why all this hatred?

        Seriously, is it envy? London is the third biggest city in Europe, so you can expect it to have a very high concentration of people...

        No, it is not envy, you are trolling and your claims have no basis at all in reality. Plenty of people have given you anecdotal evidence contrary to your claims: they are demonstrably incorrect. If you want more concrete evidence it took, oh, seconds to find this story in the local rag. Note the installation of emergency generators. Oh, and if your geography is as bad as your knowledge of emergency power provision, Penwortham is in Lancashire, not London.

        In actual fact, as I noted previously you are likely to receive a worse service in London simply because of that density you cite as an argument in your favour. The more properties affected the longer recovery target times and compensation thresholds get. This in London was a major outage - you could multiply the area by ten in rural areas and it would still be a minor outage subject to those more stringent targets.

    4. Boris the Cockroach Silver badge
      FAIL

      Idiot

      Here in Portsmouth we lost about 50% of the cities power supply due to a major sub-station fire a few years ago.

      We had the generators in place and everything back up'ish in 6 hrs.

    5. Stuart Castle

      Do you have proof that they wouldn't provide any help if it had happened outside London? Bear in mind that if UKPN and the power companies do their jobs right, most people would not even be aware the problem was serious enough to warrant bringing in generators, assuming they are aware of a problem at all?

    6. Christine Munro

      Re: "anywhere outside London"

      I live in a village somewhere in central England, and it was just our street whose power was cut off after someone helpfully drilled through a power line. In spite of the rather limited impact, within a couple of hours or so an albeit rather smaller generator had materialised to keep the lights on until they could get someone round to fix the cable.

    7. Tom 7 Silver badge

      If this happens outside london we'll get plenty of help

      all those media choppers will keep the windmills whizzing round!

      1. Danny 14 Silver badge

        Re: If this happens outside london we'll get plenty of help

        transformer blew in Carlisle and a generator was put on site, due to the geography they needed to put it on the road and had a lane closed. I saw the fire brigade in the morning and generator in the evening so they were fairly quick.

    8. SImon Hobson Silver badge

      Bringing in portable gennys is SOP for all the DNOs in the event of power outages - but what they bring in and when does depend on the nature of the fault, the customers affected, and a few other factors.

      No they aren't going to bring in a genny to support the single cottage at the end of a line if they expect the fault to be fixable fairly quickly. Yes they will bring one in when it's a few more houses and they expect it to take longer to fix. I've certainly seen small towns and villages round here with gennys parked next to the substation while they fix faults on the 11kV feeds to them (all I'll say is, cough, a little inside knowledge).

      Mind you, they can't win either way. I have friends in a village which had this treatment not long ago - genny in the field for over a week IIRC. Needless to say, some of the locals weren't grateful for having power - they were moaning about having the genny rumbling away.

      Where I used to work we had our own substation and a planned power cut while they were trimming trees. A DNO bod came to site to remove fuses and make sure we couldn't back-feed the line while they were working. I got chatting to him, and he told me an amusing tale about a local town ...

      They brought in 2 big containerised gennys to run the town while the switched off the supply lines. These had been running on no load for a while - and those who know about diesels will realise that tends to let them get cold and the upper cylinders get oily. Then they got the town's load dumped onto them, at which point the throttles opened to keep the engines at the correct speed.

      So the engines suddenly went onto load, dumping all the oily carp out the exhaust as dirty, sooty, oily smuts - just as this guys boss walked past in a nice clean hi-vis :)

    9. Colin Miller

      <quote>If this happened anywhere outside London don't expect any help whatsoever.</quote>

      Have you ever heard of Aggreko? They are one of the biggest generator hire companies, and their UK operations based in Glasgow. They provided 200MW of generators when Japan got a bit wet in 2011.

  2. Velv Silver badge

    "The lesson for anybody with a stake in their company's power supply was clear: make sure you've got a back-up plan in place and the utility or the boxed generator firm knows who they can reach if the lights do go out."

    Really the lesson is to sit down and consider what eventualities might befall your business, them develop and TEST those plans.

    Read the chapter on the New York Board of Trade in "Blueprints for High Availability" by Marcus and Stern. It makes you think not just about computers but the rest of it. How do you get your staff to a temporary office? How do you schedule their workload? How do you feed them?

    Disasters happen every day, mostly to someone else, but you never know when that knock on effect will kick in.

    1. Danny 14 Silver badge

      or just have staff who can think quick :) roof blew off our refectory building when we had really high winds a couple of Novembers ago. Quick thinking IT staff phoned the local bakery and arranged a few hundred pies to be delivered.

      1. John Brown (no body) Silver badge

        "Quick thinking IT staff phoned the local bakery and arranged a few hundred pies to be delivered."

        So, what did the non-IT staff get?

        (Also, who ate all the pies? The BOFH?)

    2. Rebecca M

      Really the lesson is to sit down and consider what eventualities might befall your business, them develop and TEST those plans.

      This strikes me as one of those hand-waving magic wand assertions by someone who thinks it sounds good but has never even put a plan in place, yet alone "tested" it.

      Pray tell, how the hell do you sensibly test a DR plan? Even the best plans will entail massive disruption to the business once a non-trivial portion of them is in effect and typically significant expense - following a genuine event business continuity insurance will kick in if obtained, not so for a test. Our plan calls for around 60% of staff to be laid off immediately in the event their premises are not available. In a disaster that is unavoidable but it is completely unacceptable for a test. Last but not least which scenario do you test? Any DR plan worthy of the name will cover a multitude of scenarios but have flexibility built in to cater for the unexpected or the particular flavour of the situation - something such as "can't use that site" sounds fairly digital in nature but simple practicalities such as "Can you still access it to retrieve equipment?" make massive differences to how the plan unfolds.

      In short a DR plan is not a simple list of things that are carried out regardless of the situation or the impact of the business but a menu of options that have some degree of preparation for implementation, the decision to execute them depends on the scenario. The idea that it can be tested and then it is proven is naive to the point even of what is in a genuine DR plan. Sure you can test specific aspects and assumptions: for example our plan depends on our "backup box"es - DJ style 18 or 20U flight cases with a disk array, a few servers, switch etc that are supposed to be able to host a couple of hundred users without them even noticing. Do they? Yes, of course, we can and have tested that. We can also check we are able to get suitable office accommodation at the drop of a hat - we do so regularly and yes we can, no need to actually hire and fit out the premises for that.

      However, the idea we are going to simulate e.g. a massive fire taking out a major office block, down to hiring an alternative, fitting out in short order, and laying off people who are not needed - that simply isn't based in reality, and would probably cause more harm to the business than doing nothing at all.

  3. Peter Simpson 1
    Facepalm

    UPS/gensets

    For those of you counting on the UPS to hold up your systems while the genset cranks over and starts up...make sure to test regularly.

    A friend in the broadcast biz had a nasty surprise when his genset didn't crank over.

    1. Martin an gof Silver badge

      Re: UPS/gensets

      A friend in the broadcast biz had a nasty surprise when his genset didn't crank over.

      When I worked at a radio station, our generator never cranked over. Until we were taken over by a bigger, more cash-rich company, our power backup consisted of a phonecall to the pager, a dash down to the studios, wheeling out the two-cylinder 10kVA Diesel generator, pulling the big "Frankenstein" switch to isolate all non-essential systems and hand cranking the beast.

      Hand cranking a Diesel is not fun. The best thing was to have someone else hold open the "cocks" on the cylinders and get up to speed without compression.

      The new owners bought an auto-start genset that could power the whole building and we tested it a couple of times a year. Glad to see the back of the Diesel to be honest.

      M.

      1. Peter Simpson 1
        FAIL

        Re: UPS/gensets

        Oh, it was *brand new* :-) And, they had "tested" it...by pushing the TEST button. Genset cranked over and started right up, just like it was supposed to.

        They hadn't wired the powerfail start-up circuits correctly. The test switch worked fine, but when it came to the Real Thing...five minutes of UPS, then crickets...

    2. bpfh Silver badge

      Re: UPS/gensets

      The RedBus datacentre went offline sometime during 2005-06 3 times in 12 hours, due to an offsite power outage, bad UPS battery management and an unplanned genny outage.

      First of all, a sparky shorted out the main power box, and tripped the whole site. Oh dear. That was corrected and the UPS fired into action and restored service.

      When the offsite outage lasted more than 30 minutes, the industrial gennys were cranked up, but they need 15 minutes to warm up before they can be attached to the internal power grid,

      With 15 minutes left on UPS power, the diesel generators took over, keeping the lights on. After 30 minutes, with reasonable load, the 3rd genny was taken offline and put back as backup, problem being that the generators could power the site, but not recharge the UPS.

      2 hours later, one of the generators failed, overloaded and tripped the second, and the UPS bank took over. The tripped genny & backup were restarted and allowed 15 minutes to sync. 2 minutes longer than the life in the UPS batteries.... Second outage.

      I'm not sure how the 3rd one happened, but I think when the external power was reapplied, somthing else went wrong and tripped the whole site again.

      End result, a lot of SDA's down, and a shedload of angry clients!

      Moral: Even with a DR in a top tier datacentre, reduncancy is king!

  4. gerdesj Silver badge

    That wasn't DR as such

    DR is "Disaster Recovery" ie what you do with your business when it suffers a disaster. Strictly speaking the disaster was for the telcos and eleccy suppliers - who then carried out their DR plan.

    What you need as a business (or whatever) is a business continuity plan which is what you do when something affects your business.

    So, backups to offsite = disaster recovery but having a genny handy in the case of a loss of power = business continuity.

  5. Anonymous Coward
    Anonymous Coward

    Solution? RBMPG (Rodentia Based Micro Power Generation)

    Any large organization should have the requisite number of hamsters, with their wheels attached to bicycle dynamos. Perhaps very large sites could use guinea pigs, I dunno, I'm no expert. I've been generating all my power at home for many years now from my gerbil, Colin. Reliable and carbon neutral. There are clearly no problems with this approach, anyone looking for problems is just being picky.

    1. Little Mouse
      Terminator

      Re: Solution? RBMPG (Rodentia Based Micro Power Generation)

      Or better still, a network (or matrix, if you will) of people. The power output of each person being the equivalent a single C-type Duracell battery.

      Mind you, you'd need to come up with some kind of distraction to keep them occupied....

      1. Danny 14 Silver badge

        Re: Solution? RBMPG (Rodentia Based Micro Power Generation)

        "Mind you, you'd need to come up with some kind of distraction to keep them occupied...."

        you'll be amazed at what threatening to add youtube, facebook, twitter etc to the blocklist can do.

      2. Anonymous Coward
        Anonymous Coward

        Re: Solution? RBMPG (Rodentia Based Micro Power Generation)

        "Mind you, you'd need to come up with some kind of distraction to keep them occupied...."

        Like TV and the Interwebs....

        1. Anonymous Custard Silver badge

          Re: Solution? RBMPG (Rodentia Based Micro Power Generation)

          Good excuse for putting in a brand new company gym, with loads of excerise bikes and such subtly connected to a few dynamos?

    2. Velv Silver badge
      Childcatcher

      Re: Solution? RBMPG (Rodentia Based Micro Power Generation)

      Your definition of the word many clearly differs from mine. Either Colin is the worlds oldest Gerbil or you've had more than one in those "many" years.

      1. Little Mouse

        Re: "Either Colin is the worlds oldest Gerbil..."

        Some misguided parents like to prevent childhood heartache by quietly replacing dead pets before the youngsters notice....

        Luckily for me my hamster has kept going for over 40 years, so my family has never had to indulge in such subterfuge.

  6. Anonymous Coward
    Anonymous Coward

    Rate this article

    4/5

    Dunno what would have made it 5/5. But 4 it is. Maybe 4.5 actually.

    Thank you.

  7. Anonymous Coward
    Anonymous Coward

    There wasn't a full DR when it happened...

    Various people have commented about incidence response times, and whether London got any kind of "special treatment", so I'm not going to comment about that. However, I think that it is worth saying something about what happened from the perspective of somebody who works in the area. The bottom line is that this was a very major incident. Generators may have been up and running quickly, but it was clearly neither quick nor easy for them to get power distributed.

    I work in Covent Garden, about five minutes walk from the Kingsway where the fire happened. Fortunately we were okay when it happened. However, power was off on the other side of our street and beyond, meaning that lots of cafes, restaurants, theatres, offices, etc. etc. were cut off.

    The power was *not* back on a few hours later or even the next day.

    The fire happened on Wednesday 1st April. The next day, power was not restored in Covent Garden (I had a look around the area starting with the Royal Opera House, north up to Long Acre, and east across to the Kingsway). On Friday 3rd (Good Friday) power was still off. Monday (6th) was Easter Monday. On Tuesday (7th) power was restored to many places, although a lot of cafes and restaurants were still closed. I do recall heading over to my usual place for lunch on Tuesday and finding them with power only in half their shop. I think that power seemed to be fully restored (at least to the places local to me) on Wednesday 8th.

    For information, they have been reopening affected sections of the northbound side of the Kingsway over the last couple of weeks, and I think that it may now be open to northbound traffic all the way from the Strand. However, we are two months from when the fire happened, so clearly this was a very major incident.

    1. ilmari

      Re: There wasn't a full DR when it happened...

      i think the point of the article was that those businesses with a plan and contract were sent generators within 25 minutes. The lazy businesses took longer because they had to figure out first how big a generator they need, where to park it, and how to hook it up, etc.

      Presumably there was a third class of people/businesses that never ordered generators and just waited for grid power.

  8. Kubla Cant Silver badge

    Where?

    I wouldn't describe Kingsway as "central West London". That sounds more like Hammersmith.

    Depending on your criteria, Kingsway about as central London as you can get. Bank is probably the centre of the Roman city, but Aldwych, at the bottom of Kingsway, is named after Lundenwic, the Saxon trading settlement.

    Was the expression "central West London" derived from the fact that Kingsway has a WC2 postcode?

  9. Anonymous Coward
    Anonymous Coward

    Black starting...

    You'd be amazed how fast you can organise a generator - when the power cable to the top end of Wasdale failed catastrophically there was a generator set up there and installed quite quickly - although it was connected using giant crocidile clips to the transformer outputs which looked a bit of a bodge.

    If you have a real infrastructure outage the priority isn't on getting generators up its a black start of the grid. Look for your nearest hidden IT bunker that has a generator set onsite and lots of diesel - you might find one with direct HV tie lines to a power station and they are the places that get all the attention as they are what you need to restart a baseload power station.

    AC obviously so someone doesnt come and pinch all our black start diesel supplies...

  10. Anonymous Coward
    Anonymous Coward

    I walked through the area when it was all happening. Generators humming away absolutely everywhere, quite impressive really.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019