back to article Fingers crossed tomorrow morning for Telecity's third repair shot

UK internet hub Telecity on Thursday plans its third shot at fixing a power systems problem that fried customers' servers. According to an email seen by The Reg, Telecity’s chiefs have scheduled a three-hour window between 01.00 – 04.00 GMT on 26 November to fix wonky power phasing at the firm's Sovereign House data centre. …

  1. nbc

    Impressive...

    "Engineers will use their repair window to rectify the power-phasing problem, which emanates from a transformer that was accidentally produced by an engineer on the afternoon of 17 November."

    I keep reading this sentence and I'm still none the wiser. Did an engineer really produce a transformer?

    Which one?

    1. Vince

      Re: Impressive...

      Obviously they couldn't tell you which transformer. That's the problem you see. Damn those robots in disguise!

    2. Bob H

      Re: Impressive...

      Sounded painful to me, I've taken a few zaps in my time but none has resulted in me emitting anything that sharp.

      1. CrazyOldCatMan Silver badge

        Re: Impressive...

        > I've taken a few zaps in my time

        I suspect said engineer is currently acting as the main busbar for the power feed. Carbon works quite well y'know.

        1. Bob H

          Re: Impressive...

          I wonder what colour temperature said carbonised human is currently at....

    3. I. Aproveofitspendingonspecificprojects

      Gizza job

      "Engineers will use the repair window, some time in November, to rectify the power-phasing problem with a transformer."

      Think of the ink you will save.

  2. Anonymous Coward
    Anonymous Coward

    Design

    It would appear the first and biggest problem is the design of the data centre power supplies.

    A diesel generator is not a UPS, it takes time to start and needs synchronising with what is powering the place before it can take over. If it isn't then you can get very strange waveforms appearing at the outputs of transformers.

    It sounds as if they have IT people trying to solve a problem that requires electrical engineers to correct. If they are not employing electrical engineers then I can foresee this rolling on until such time as they do.

    1. Vince

      Re: Design

      They're using DRUPS units - they're not *just* Diesel generators, they're combined UPS and Generators - so the UPS bit is either battery or flywheel (I imagine battery at Sovereign House, but I could be wrong), with a Diesel Generator to follow.

      That said, Sovereign House has had power issues on more than one occasion before, and we moved kit out after the last farce.

    2. Anonymous Coward
      Anonymous Coward

      Re: Transformers?

      "strange waveforms appearing at the outputs of transformers."

      What IT kit built in the last couple of decades has a transformer in it (or at least a 50 or 60 Hz transformer)?

      Switched mode power supplies rule the modern world. Transformers in equipment power supplies are pretty much extinct (cost too much vs an SMPS). Many switched mode power supplies don't seem to be hugely fussed what you feed them on, but allegedly various bits of kit in the datacentre in this picture have been "destroyed by surges"?

      Now if you're talking 11kV to 440/230V transformers that's a different kettle of worms.

      1. Anonymous Coward
        Anonymous Coward

        Re: Transformers?

        The problem there is that almost all IT kit (except in specialist aeronautics/military stuff) at some point is designed to rectify 50/60Hz mains and smooth it to high voltage DC before feeding it into the switch mode PSU (which still uses a transformer albeit at many tens if not hundreds of kilohertz up into the MHz range)

        So 'spiking' the mains will destroy the sacrificial components, filter capacitors, VDRs, Thermistors etc. that are designed in to prevent that spike getting through the PSU to smoke your business critical servers.

        An 'odd' waveform could potentially mean many times the rated input voltage for a PSU to deal with, at best it will shut down, at worst, you have a data centre full of leaking magic smoke.

        1. Sgt_Oddball

          Re: Transformers?

          Its really fun if the spike manages to pop a component too. Having had my desktop psu recently go pop rather than the rest of the rig I'm rather greatful for this sort of capability.

          Would be interesting to see these sort of tests being included in benchmarks so you know if the kits upto spikes and by how much.

      2. Anonymous Coward
        Anonymous Coward

        Re: Transformers?

        "Switched mode power supplies rule the modern world."

        Only up to a certain size. This is a data centre, so power requirements are large. Big 50/60Hz transformers have a reliability advantage over SMPS, as they don't pop so easily due to voltage spikes (speaking as someone who designed SMPS in a past job).

        1. Anonymous Coward
          Anonymous Coward

          Re: Transformers?

          "Big 50/60Hz transformers have a reliability advantage over SMPS"

          May well be so (they certainly win on simplicity, component count, etc), but I also know that SMPS are cheaper and therefore ubiquitous. What modern kit still has a 50Hz transformer in it? Examples very welcome.

          I much prefer the suggestion that the PSU front ends have had mains-side sacrificial components blow. The idea that this has "destroyed servers" is a bit hard to take, especially given that any decent server will likely already have customer-replaceable PSUs. Unless someone's got the wiring far more seriously wrong than we've been told so far.

  3. Commswonk

    Re: Design

    A diesel generator is not a UPS, it takes time to start and needs synchronising with what is powering the place before it can take over. If it isn't then you can get very strange waveforms appearing at the outputs of transformers.

    Well it can be if the diesel engine is part of an integrated design; I know of systems (and not recently) with an AC Motor, a DC motor driven by BIG batteries, and alternator (all on a common shaft, or at least shafts coupled in line, plus a diesel generator. The incoming AC drives the motor which it turn drives the alternator plus the DC motor; if the incoming supply fails the battery supplies the DC motor until the diesel generator self - starts and then takes the load and drives the AC motor again. ISTR other systems whereby instead of a DC motor there is a BIG flywheel and a clutch between the "common" shaft and the diesel engine in which the flywheel (with a large mass) drives the common shaft for long enough for the engine to start. The start and run - up time (after which the clutch couples the two shafts) can be as short as a few seconds if the system design includes keeping the engine coolant warm so that it isn't starting from true "cold". Getting it in - phase with the incoming supply isn't required because, er, there isn't one.

    OTOH if they are trying to engineer a "no - break" changeover between one supply and another derived from a generator in the car park then getting the phases in sync can be rather important. If that is the case then It sounds as if they have IT people trying to solve a problem that requires electrical engineers to correct. If they are not employing electrical engineers then I can foresee this rolling on until such time as they do may well turn out to be true. Synchronising a generator in one place with an incoming supply in another isn't impossible but it can be messy.

    I don't want to be gratuitously critical but I found this article (and that which preceded it) a bit unclear for possibly the same reason - perhaps written by an IT guru rather than a power systems specialist. Articles about electrical problems should not have the word phase in them other than in reference to supply phases; "stage" is much better because it cannot be confused with electrical phase.

    1. Doctor Syntax Silver badge

      Re: Design

      "perhaps written by an IT guru rather than a power systems specialist"

      Or maybe rewritten from a press release written by a PR man who understood neither?

    2. tony2heads

      Re: Design

      I believe that there are rotary UPS systems like the system that you describe with zero start up time (or less than 1 cycle of 50Hz). The diesel is kept warm and with a BIG flywheel the clutch auto engages when incoming power drops.

      But these are expensive, large and very hard to retrofit to a building.

      1. Commswonk

        Re: Design

        IIRC there are, but the "1 cycle of 50 Hz" might be a bit ambitious. If the shaft speed (and therefore the diesel engine speed) is 3000rpm* in 1 cycle of 50 Hz - which equates to 20 mS - the engine will have turned once so assuming it will have "fired" in that time might be a bit much. Having said that 3000rpm is rather too fast for a "big" engine so 1500rpm** might be more likely. Bigger engines still will not run above 500rpm*** or thereabouts without self - destructing, but they are so big that electric starting isn't an option - they commonly used compressed air and are certainly not fast starting; even a "flywheel start" is probably not practicable.

        I have a vague recollection that systems like that where there is a very fast run up require not only the cooling system being kept warm but the lubricating oil being pumped more or less continuously to ensure that all the bearings are coated; conventional lubrication where the engine just pumps its own oil will not distribute it fast enough to prevent a catastrophic failure.

        * 2 pole alternator; let's not worry about 3 phases for this discussion.

        ** 4 pole alternator; ditto!

        *** I know; 500rpm is not much good if you want 50 Hz. :)

  4. nsld
    Coat

    Have they tried

    Turning it off, then turning it back on again?

  5. Kernel

    RE: Have they tried

    Living at the other end of the world I haven't been following the saga in detail, but I get the impression they've got the turning it off bit sorted - it's the next stage they need to work on now.

    1. allthecoolshortnamesweretaken

      Re: RE: Have they tried

      Even the "turning off" bit can be pretty tricky sometimes.

      Complex systems, "engineers" that are really technicians with a two week orientation under the belt - what could possibly go wrong?

      1. Anonymous Coward
        Anonymous Coward

        OT re Berlin Airport

        "Even the "turning off" bit can be pretty tricky sometimes."

        Thanks for that link.

        I used to follow the Berlin International Airport saga for a few years when I used to fly out there once a year but haven't done so for the last couple of years. I hadn't realised that it was still ongoing. Nice to see that even the Germans can have a Terminal 5 experience (but much much bigger) when the conditions are right. Or maybe not so nice if you're one of the ones paying for it or waiting for it.

        http://www.aeronewstv.com/en/transport/airports/2861-berlin-international-airports-construction-site-fiasco.html

        "Complex systems, "engineers" that are really technicians with a two week orientation under the belt - what could possibly go wrong?"

        Apparently quite a lot, judging by the airport story.

        1. I. Aproveofitspendingonspecificprojects

          not to mention cases of corruption

          Who told you not to mention the cases of corruption?

  6. jaywin

    I hear there was another power cut overnight.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like