back to article I don't know but it's been said, Amphenol plugs are made with lead

Tear your gaze away from the retreating rear of the weekend and instead look forward to another week filled with opportunity, adventure and, of course, The Register's Who, Me? Column. Today's cock-up comes courtesy of reader "Terry", who told us of his time working in a data centre that served a number of high-profile …

  1. Pascal Monett Silver badge
    WTF?

    "The router went dark"

    Why ? There is nothing that tells me what happened, except that the cable apparently went "between the guard rails". And ? Did it hit the circuit board ? Did it somehow fall in the power supply, making a short-circuit ?

    Could someone please explain why a cable falling on a router makes it "go dark" ? I don't have a clue why this happened.

    1. Solarflare

      Re: "The router went dark"

      From earlier in the article...

      Further, the supplies are recessed slightly so objects dropped from above cannot hit the power switches

      So i think it's heavily implied that the heavy cable hit the power switch.

      1. Jason Bloomberg Silver badge
        FAIL

        Re: "The router went dark"

        Further, the supplies are recessed slightly so objects dropped from above cannot hit the power switches

        Maybe they should have gone for a toggle switch with a flip-up cover which means it can't be activated if anything does hit it, and it it has to be flipped-up to be activated?

        You know; the sort of thing the military use all the time to stop switches and 'bombs away' buttons being activated accidentally.

        Halfords, Amazon and eBay stock them. Any decent elctronic component supplier does and they cost all of a couple of quid.

        It seems my Raspberry Pi reset switch is better protected than that router.

        1. Louis Schreurs

          Re: "The router went dark"

          When I build my on-stage guitar amps I put the Main switch so that if struck from above, the amp will switch ON.

          1. jake Silver badge

            Re: "The router went dark"

            "if struck from above, the amp will switch ON."

            Experience suggests that in that case, someone will manage to snag the switch with a 1/4 inch jack plug dragging a cable over the amp, and thus pull it up and off.

      2. Mark 85 Silver badge

        Re: "The router went dark"

        The article also states that the turned the router back on. So a switch hit is obvious.

    2. jake Silver badge

      Re: "The router went dark"

      An RJ21 connector swung down and hit the power switch, probably on a Cisco 7000 of some description (the prior AGS and AGS+ couldn't switch at the rate mentioned). Some folks had (optional) protective bars fitted, in the hopes of avoiding this kind of accident.

      Murphy was an optimist. (If you're British, substitute Sod or Finagle for Murphy.)

      1. Doctor Syntax Silver badge

        Re: "The router went dark"

        "If you're British, substitute Sod or Finagle for Murphy."

        Murphy is alive and well and living in Britain. But Finagle? I thought he was one of yours.

        1. hmv Bronze badge

          Re: "The router went dark"

          They swap places every so often just to keep things confused.

          1. A.P. Veening Silver badge

            Re: "The router went dark"

            I am just surprised Muphry hasn't made an appearance yet in this discussion ;)

            1. Doctor Syntax Silver badge

              Re: "The router went dark"

              It has.

              1. dfsmith

                Re: "The router went dark"

                I dont' see any-one whose mentionned Muphry in the preseeding tecks.

                1. Doctor Syntax Silver badge

                  Re: "The router went dark"

                  Whoosh.

                2. JJKing Silver badge
                  Thumb Up

                  Re: "The router went dark"

                  I don't' see any-one whose mentioned Murphy in the preceding tecks.

                  Just look 4 posts up from your one.

                  Typos fixed :-)

        2. jake Silver badge
          Pint

          Re: "The router went dark"

          I stand corrected. I knew John Campbell used it, but I didn't realize he invented it. I always thought Finagle was one of yours because I've never heard it "in the wild" over here (outside of Astounding, of course), but I have heard it over there.

          Likewise, when I used Murphy in Yorkshire way back when I got odd looks, but I soon found that Sod was the goto in such situations. Strange how epigrams vary so much from region to region. Noted, and ta.

          Beer. Seems like a good way to start the week :-)

          1. Doctor Syntax Silver badge

            Re: "The router went dark"

            "Likewise, when I used Murphy in Yorkshire way back when I got odd looks"

            Maybe they were just giving you odd looks on principle.

            Over here we also use Murphy to trap unwary seekers after umbrage. A comment such as "Murphy have dug up the road again" is apt to flush out those who take umbrage on behalf of Irish labourers. Then we can point out that the Murphy Group are well known public infrastructure contractors.

            1. Anonymous Coward
              Anonymous Coward

              Re: "The router went dark"

              Maybe they were just giving you odd looks on principle.

              That was a good one for a Monday, thanks :)

            2. Antron Argaiv Silver badge
              Alert

              Re: "The router went dark"

              ...well known public infrastructure contractors.,

              I believe that particular type of circuit disruption is referred to in the trade as "backhoe fade", at least in the US

              1. jake Silver badge

                Re: "The router went dark"

                Backhoes only fade when they don't have enough fiber in their diet.

                1. -tim
                  Alert

                  Re: "The router went dark"

                  You know that is ISO layer Zero. Just like the number of bits that pass after the event.

              2. eldel

                Re: "The router went dark"

                Oh I had one of those. Back in the late 80s - was demonstrating the new Telegram Retransmission Switch to BTI to get their sign-off. Was based on a Sequoia fault-tolerant unix box. We had the test traffic running as we were casually pulling out circuit boards and disks - all that happened was that it slowed down. We were just getting to the end of the demo when Paddy and his JCB took out power to the whole building - and the demo system wasn't on the UPS.

                Underwear changes all round. Fortunately the BTI guys hung around for long enough for the power to be restored. At which point the whole thing just carried on where it left off. The force of exhaled breath probably started a major meteorological perturbation..

              3. MJI Silver badge

                Re: "The router went dark"

                I only recently found out what a backhoe was.

                They are JCBs over here.

                1. jake Silver badge

                  Re: "The router went dark"

                  JCB manufactures heavy equipment (amongst other things).

                  One such piece of heavy equipment is called a backhoe.

                  Other manufacturers of heavy equipment also make backhoes.

                  A friend "over there" has a Case 580N backhoe. She doesn't call it a JCB.

                  1. MJI Silver badge

                    Re: "The router went dark"

                    We call them diggers. Or after the UK market leader JCB.

                    They did the first dedicated tractor style one or as well call them JCB.

                    From Wiki

                    In the same year, JCB launched the first European hydraulic loader, followed by a backhoe with a 180° slew, fitted to a tractor, in 1953. In 1957, CASE developed the first integrated tractor backhoe loader at the Case New Holland plant in Burlington, Vermont, helping contractors complete their work more efficiently.[8] JCB introduced the first dedicated backhoe loader, incorporating the excavator and the major loader in a single, all-purpose tool. [9]

                    Because of the long-time predominance of the JCB marque in the United Kingdom and Ireland, it has become a genericized trademark there, and backhoe-equipped diggers are commonly called JCBs, while the term "backhoe" as an excavator component is almost unknown to the general public in this context. The founder of the JCB company, Joseph Cyril Bamford, is the only Briton to be honored in the Association of Equipment Manufacturers Hall of Fame.[11]

                    There is a book about them.

                    https://en.wikipedia.org/wiki/The_Nome_Trilogy#Diggers_(1990)

            3. jake Silver badge

              Re: "The router went dark"

              "Maybe they were just giving you odd looks on principle."

              Those looks I was used to, being a Californian schoolboy in Yorkshire. These looks were different.

        3. Sequin

          Re: "The router went dark"

          I have never heard Finagle used in the context of Murphy's/Sod's law though I have often applied the "Finagle Factor" - that value which when added to, subtracted from, multiplied by or divided into the value you have, gives you the value you should have got.

        4. Martin Gregorie Silver badge

          Re: "The router went dark"

          I don't remember Finangle having anything to do with the sort of event masterminded by Murphy or Sod.

          Finangle invented the small, variable constant used by chemistry and physics undergrads to make the result of a lab experiment match the required result.

          1. Terry 6 Silver badge

            Re: "The router went dark"

            And I used it to make sure that the totals for number of clients recorded as seen=total for all staff members records= total male+female=total for each location=total of all ethnic categories. Because it never did otherwise. Not once ever.

            The trick was to find two that did match and adjust the others proportionately.

          2. Baroda

            Re: "The router went dark"

            ' to make the result of a lab experiment match the required result.'

            <sigh> I remember rubbing out the actual values I had measured and fudging them in my pencil-on-graph paper curve to make the experimental results fit the chemistry book 'S' curve. As had legions of children before and since no doubt.

            The experiment was dropping zinc granules into HCl and graphing the release of hydrogen - starting slowly, rapidly reaching a steady high rate then dropping off to zero as the Zinc was exhausted.

            The slow-down I had measured and eliminated midway or higher on the steepest part of the graph was no more.

            Many years later I heard of a truly scientific chemistry teacher observing this (probably having ignored it before) and carefully re-doing the experiment. The rate *did* slow then speed up.

            It turns out that the hydrogen bubbles forming on the surface of the zinc and clinging by surface tension before floating off prevent the maximal access of the HCl to the surface as the reaction progresses. (My best understanding - if anyone can improve or correct this please do.)

            Hat's off to that anonymous teacher.

            1. Must contain letters

              Re: "The router went dark"

              I think this actually featured on a bbc program called something like young scientist of the year at some point in the late 60s/early 70s.

        5. MonkeyCee Silver badge

          Re: "The router went dark"

          Last I checked Murphy appeared to be running most of the bloody country. Certainly the political and financial sectors.

          Fairly sure the M in MoD is for Murphy.

      2. Korev Silver badge
        Joke

        Re: "The router went dark"

        Come Brexit the UK can throw off the shackles of EU regulations like Murphy's law

        1. Benson's Cycle

          Re: "The router went dark"

          Sometimes a Murphy Event is so huge and catastrophic that it leaves an event horizon from inside which, if something goes wrong, no information about it can reach an observer.

      3. theblackhand Silver badge

        Re: "The router went dark"

        My recollection of the Cisco 7000/7500/7600 is that they either have hardwired DC PSU's with no power switch or a "clunky" turning knob type power switch to power them on.

        Having said that, Cisco produced so many different PSU options for these beasts that there is likely something exactly as OP described.

      4. The_FFrey

        Re: "The router went dark"

        We know about Murphy's Law in the UK, but we usually refer to Tuckers Law.

        It's much more British

        1. The Mole

          Re: "The router went dark"

          We do? First time I've ever heard of it, as far as I'm aware its Sods law or Murphy's law. Sod's law being you'll get the worst option, whilst Murphy's law is what can go wrong will go wrong at the worst possible time, and even if it can't go wrong it will anyway..

      5. Muppet Boss

        Re: "The router went dark"

        Ah those infamous Cisco 4500-E super sensitive power switches. They could flip by simply breathing at them. Cisco 6500-E PEM power switches were a way more secure and better designed.

        https://www.cisco.com/c/dam/en/us/td/i/000001-100000/75001-80000/79001-80000/79138.ps/_jcr_content/renditions/79138.jpg

    3. Korev Silver badge
      Joke

      Re: "The router went dark"

      Could someone please explain why a cable falling on a router makes it "go dark" ? I don't have a clue why this happened.

      Easy, it blocks out the sunlight...

    4. fobobob

      Re: "The router went dark"

      Based on descriptions, it was likely a Cisco 6513. I suspect they may be referring to the handles on the power supplies as guard rails.

      1. Muppet Boss

        Re: "The router went dark"

        >Based on descriptions, it was likely a Cisco 6513. I suspect they may be referring to the handles on the power supplies as guard rails.

        Nope Cisco 6500-E had very difficult to turn off accidentally power switches, 1950's-style. This surely was a Cisco 4500-E switch which power supplies had a very sensitive power button only protected by a thin barrier on each side which protected nothing - without any sort of a plastic cap as on some other models. It could flip by literally knocking on the PSU.

  2. DougS Silver badge

    So why did it have a dead power supply?

    Did the DoD not think this critical router was worth having a spare power supply on site so they were waiting for Cisco to show up with a replacement? Was it not properly monitored so he only found out it had a dead power supply after? Or did he know had a dead one, but was not authorized to touch it, and whoever who was authorized was not available on site yet?

    it doesn't do much good to have redundant hot swap power supplies if you knowingly let a device operate with a dead one. Maybe they need three power supplies to handle such a scenario!

    1. Chris Miller

      Re: So why did it have a dead power supply?

      I thought this was going to be similar to some of my war stories. For redundant systems, the biggest cause of dual failure is an engineer arriving with a replacement unit and then trying to swap out the good one instead of the failure.

      But in one machine room I was responsible for, we had the usual three-phase power supply and the critical kit had dual power supplies. At one point we lost a phase and no-one noticed, except for a group of servers (not my responsibility, thankfully) whose dual power supplies were both powered from the same phase.

      1. brotherelf
        Facepalm

        Re: So why did it have a dead power supply?

        Been there, seen that. Also, rack full of dual-PSU servers, all connected to a single-PSU switch.

        1. John Brown (no body) Silver badge

          Re: So why did it have a dead power supply?

          "Been there, seen that. Also, rack full of dual-PSU servers, all connected to a single-PSU switch."

          To be fair, that really depends on what you are protecting against and/or the available budget level.

          1. Anonymous Coward
            Anonymous Coward

            Re: So why did it have a dead power supply?

            To be fair, that really depends on what you are protecting against

            In most cases that turns out to be accountants who only see a now value when you're talking about spares, and not the cost of downtime..

            1. Antron Argaiv Silver badge

              Re: So why did it have a dead power supply?

              You can't protect against accountants.

              1. eldel

                Re: So why did it have a dead power supply?

                Well, theoretically you can - but the relevant hunting licence is almost impossible to get nowadays,

            2. Alan Brown Silver badge

              Re: So why did it have a dead power supply?

              "In most cases that turns out to be accountants"

              There are generally (BOFHish) ways of making accountants realise the value of spares, and redundant power.

        2. fobobob

          Re: So why did it have a dead power supply?

          Also fun: dual-psu servers connected to a single power source with cheap Y-cables.

          1. DougS Silver badge

            Re: So why did it have a dead power supply?

            Well at least that protects against failure of the power supply. More than a few data centers in the US don't have redundant circuits to the individual cabinets. Still should have two separate cords though, so accidentally unplugging the wrong cord from the PDU doesn't kill both PSUs.

            1. fobobob

              Re: So why did it have a dead power supply?

              Very true. In my specific case, our datacenter had intermittent issues with the plugs slowly creeping apart, causing power failure warnings and more rarely total dropout of the server if this wasn't caught. After requesting some photos for an unrelated issue, we noted the cables to be hanging under their own weight; guess they hadn't discovered tie straps. They did eventually outfit the racks with additional PDUs, but I don't think they ever pursued redundant circuits.

              1. Alan Brown Silver badge

                Re: So why did it have a dead power supply?

                "our datacenter had intermittent issues with the plugs slowly creeping apart"

                This is why I use _locking_ sockets wherever possible (the kind which anchor onto the plug's earth pin and don't require any uniquely special plug)

        3. Jens Goerke

          Re: So why did it have a dead power supply?

          BTDT, albeit with a SAN switch - one PSU failed and took the fuse with it, so the other PSU ended up without power as well, taking out half of the SAN. That happened on a Friday evening with me on call, but no call went out.

          Monday morning the fault was discovered, the only effect being that a multiple-hour long billing run during the weekend had taken a few minutes longer than usual.

          The SAN had been planned and built properly, with redundancy in mind, so the second half still worked flawlessly and the PSU could be replaced and rewired to another fuse/phase without undue haste.

    2. jake Silver badge

      Re: So why did it have a dead power supply?

      I can't tell you how many times I've run into places where the PTB would happily spring for the redundant power supplies to power the kit, but would refuse to pay for actual on-site spares. Yes, including in .gov and .mil installations.

      I didn't say it made any sense, I'm just reporting what I've seen.

      1. John Brown (no body) Silver badge

        Re: So why did it have a dead power supply?

        Sometimes monitoring and/or alerts are missed or mis-configured too. I got sent out to a data centre to swap an HDD for, erm, a well known World-wide Payments system. The HDD was obviously part of a RAID. It had two hot spares, one of which had already failed over correctly to replace the faulty drive. All good so far. But when I got there, as instructed, called the admin who flashed the LED on the faulty drive as per procedure to make sure I got the correct drive (obvious anyway as it was the only drive with a red fault LED showing rather than green ready/busy), I mentioned that there was another red flashing LED, this on one of the two redundant PSUs. Cue a short silence followed by the tappity-tappity of a keyboard and a "Are you sure? There's no logs or alerts showing". They sent me back the next day to swap out the PSU and confirmed the logging had reported the swap out correctly.

    3. Doctor Syntax Silver badge

      Re: So why did it have a dead power supply?

      "Maybe they need three power supplies to handle such a scenario!"

      Or four when for when two have been left unswapped. There's no end to the penny-wise scrimping of bean-counters when it comes to postponing essential servicing.

      1. Terry 6 Silver badge

        Re: So why did it have a dead power supply?

        There's a psychological weakness in all redundant equipment systems. When one goes over the knowledge that there's another still working means that no one gets too bothered. It's tomorrow's problem and we all know that tomorrow......

    4. tip pc Bronze badge

      Re: So why did it have a dead power supply?

      in the dim distant past we normally ran 2 x routers with 2 x power supplies in each to be sure things run even if the psu's stay up but the router dies or someone applies a stupid config and kills 1 of the boxes.

    5. Nick Kew Silver badge
      Facepalm

      Re: So why did it have a dead power supply?

      Did the DoD not think this critical router was worth having a spare power supply on site so they were waiting for Cisco to show up with a replacement?

      Who needs a spare when you've got a contractor to blame?

    6. phuzz Silver badge
      Stop

      Re: So why did it have a dead power supply?

      "Maybe they need three power supplies"

      That won't always help. We use several HP Blade enclosures at work. Now, these enclosures can take up to six hot-swap power supplies, and depending on how many blades you're running (etc.) it'll run on only two.

      I was busy cabling in a new UPS, and also using the opportunity to tidy some cabling. To do this, I was carefully checking that various machines had redundant power, and then moving a single power cable at a time. This was going fine until I reached the blade enclosure which was running on four PSUs, however, as soon as I removed one, the whole thing went dark, taking half a dozen important customer servers with it.

      It eventually turned out that one of the power supplies was faulty, and when the blade enclosure tried to draw more than a few hundred Watts, it failed, putting too much load on the remaining two supplies which also tripped.

      These days that enclosure has five redundant PSUs, and they're all the high wattage ones, so it should be able to cope with two (fingers crossed).

      1. Mark 85 Silver badge

        Re: So why did it have a dead power supply?

        These days that enclosure has five redundant PSUs, and they're all the high wattage ones, so it should be able to cope with two (fingers crossed).

        "Should be" is a pipe dream... a faint hope. If it's not been tested, assume the whole thing will fail. Finding this out the hard way is a bitch. Don't ask how I know.

        1. A.P. Veening Silver badge

          Re: So why did it have a dead power supply?

          Don't ask how I know.

          No need to ask when you phrase it like that, you got acquainted with at least one of my favourite female canines. The next question though is whether it was Karma or her sister Kismet?

      2. DougS Silver badge

        Testing testing testing

        I always tell clients that they need to schedule regular tests of any redundancy (both physical and logical) they depend on, because the tests done once when things are installed right before going into production aren't sufficient. ESPECIALLY where software is concerned i.e. clusters, but hardware tests of redundancy where feasible is a good practice if you are already taking down the hardware for regular patches, configuration changes, etc.

        Some are smart enough to schedule the time, others aren't. You can only tell them, you can't make them perform the tests regularly.

        When I've worked with HP blades before they seemed very good about reporting module failure, so I'm surprised that failed power supply wasn't alerted previously. Or perhaps it was and it was sitting in someone's queue ("no need to rush on this because it still has power")

        Having the ability to have a lot of power supplies, of different types, to meet a wide range of needs is nice but the monitoring needs to be able to tell you how much redundancy is left so replacement can be properly prioritized.

        1. phuzz Silver badge

          Re: Testing testing testing

          The faulty PSU reported as good in every possible way, even after it had failed. I think it was capable of producing a small amount of power (1-200W), but as soon as it had any big load it just died, in such a way as to kill the whole enclosure.

          I tried it in another enclosure afterwards and it still showed as good, and working. IIRC we did get a free replacement from HP though.

        2. Jonathan Richards 1
          Thumb Up

          Re: Testing testing testing

          +1, I call this the Magneto Principle. Light aircraft used to have (maybe still do, I don't know [1]) the very simplest engine equipment, because: fewer things to go wrong. For ignition there would be two redundant magnetos to generate the HT spark. Part of pre-flight would be to run up the engine and switch off each magneto in turn, to ensure that they were both working well enough to support ignition independently.

          [1] It seems unlikely, though. The first pageful of search results for 'magneto' all refer to an X-Men comic character of whom I've never heard. Mumble grumble.

          1. Anonymous Coward
            Anonymous Coward

            Re: Testing testing testing

            My mower (mid-90s?) uses a magneto for the spark. (It also has a separate alternator for charging the battery.) The only part of that circuit that is ever likely to fail would be the spark plug itself. Simplicity is a wonderful thing.

            When I first got it, used, I spent some time trying to figure out why it wouldn't turn off when the key was removed. Apparently turning the key to "off" connected a lower-voltage wire from the magneto to ground, preventing spark - and that wire was disconnected. Pulling the spark plug wire with the engine running was definitely no fun.

            1. jake Silver badge

              Re: Testing testing testing

              In my experience, when it comes to lawn mower ignition, a couple of things are more likely to quit before a quality spark plug. The first is generic corrosion of the wiring caused by dis-similar metals exposed to moisture. The second is work-hardening of the copper wiring in the coil(s), leading to breakage. The third is timing issues caused by hitting a large object with the blade, bringing everything but the flywheel to a sudden halt. Momentum keeps the flywheel moving, which bends or breaks the key that keeps the crank & flywheel positioned properly with respect to each other. The forth is the magnet(s) in the flywheel becoming dislodged by the same sudden stoppage (rare, becoming more common, see below). Lastly, the magnets can actually lose their magnetism over time (I saw it only once before the rise in Chinese motors, now I see it a couple times per year).

          2. jake Silver badge

            Re: Testing testing testing

            Yes, light aircraft still run dual magnetos, and any competent pilot will check both before flight.

  3. LDS Silver badge

    Wasn't the military reason to create an "arpanet" to obtain redundand paths to reach a destination?

    And a single router outage caused so much trouble? Maybe they should have designed their network better, if it was so critical?

    1. hmv Bronze badge

      Re: Wasn't the military reason

      What makes you think it was a single link?

      If there was a redundant link, it would be connected to a different router in a different DC with the dark fibre travelling through entirely different paths.

    2. jake Silver badge

      Re: Wasn't the military reason to create an "arpanet" to obtain redundand paths

      No. The (D)ARPAnet was just a research network designed to research networking. Redundant paths were included for one simple reason: The hardware of the day was flaky.

      .mil already had SLFCS and then MEECN for the kind of thing you are suggesting.

      1. Anonymous Coward
        Anonymous Coward

        Re: Wasn't the military reason to create an "arpanet" to obtain redundand paths

        I sincerely hope that MEECN was/is pronounced "Mekon". Too good an opportunity for punnery to resist!

    3. swm Bronze badge

      Re: Wasn't the reason to create an "arpanet" to obtain redundand paths to reach a destination?

      There were two separate routes from east to west in the original ARPANET. A north and a south one. A single back hoe took them both out as the phone company routed both east/west links through a single cable. That was when they discovered that it is important to specify different physical routes.

      1. jake Silver badge

        Re: Wasn't the reason to create an "arpanet" to obtain redundand paths to reach a destination?

        News to me ... I remember three links as early as '73 ... Rand/USC to the Pentagon, Stanford to Harvard, and Berkeley to MIT ... Each link made several stops along the way, and there were several mid-continent cross-connects. All three of the west coast schools were connected directly.

        That said, in late 1977 I managed to take down all the PDP10 kit at Stanford and Berkeley with a software upgrade. Effectively split the West coast ARPANet in half for a couple hours. Not fun having bigwigs from Moffett and NASA Ames screaming because they couldn't talk to JPL and Lockheed without going through MIT ...

  4. Moosh
    Coat

    Sounds like he was given the wrong type of ladder

    He may be entitled to compensation

  5. chivo243 Silver badge
    Facepalm

    keeping swappable spares in stock might make sense

    Been down this road... and back up it a few times. No spare for the gear, it went down, howls all around. When we replaced it, bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf. These are the costs that drive my manager batty... That said, we've not has a PS die in many a year. Luck?

    1. Mongrel

      Re: keeping swappable spares in stock might make sense

      Luck?

      Murphy.

    2. katrinab Silver badge
      Boffin

      Re: keeping swappable spares in stock might make sense

      No. The mere act of having a spare in stock prevents it from breaking.

      1. Chris G Silver badge

        Re: keeping swappable spares in stock might make sense

        The ' Better to have it and never need it' rule works for just about everything.

        1. Chris King Silver badge

          Re: keeping swappable spares in stock might make sense

          That's the "Swap out the dud and beat Murphy round the head with it" strategy.

      2. Paul Shirley

        Re: keeping swappable spares in stock might make sense

        Too true. 15+ years back I half filled a shelf with variable wall warts after a run of failures. Since then the devices have always died before their psu's!

      3. vtcodger Silver badge

        Re: keeping swappable spares in stock might make sense

        The mere act of having a spare in stock prevents it from breaking.

        Unless the spare is itself defective. In which case the probability of failure on the Friday evening of any randomly selected holiday weekend is very close to 100%.

        1. Anonymous Coward
          Anonymous Coward

          Re: keeping swappable spares in stock might make sense

          Worse, the defect could result in a cascade that takes out ALL standing supplies at once. IOW, in trying to fix it, you actually blow the whole works up.

      4. Anonymous Coward Silver badge

        Re: keeping swappable spares in stock might make sense

        "The mere act of having a spare in stock prevents it from breaking."

        Until you start taunting Murphy with your 'insurance' unit, at which point he vows vengeance and will ensure that you have 2 failures in short succession.

        1. A.P. Veening Silver badge

          Re: keeping swappable spares in stock might make sense

          That isn't Murphy, but his cousin Jinx, who shows up for really heavy cases ;)

        2. kain preacher Silver badge

          Re: keeping swappable spares in stock might make sense

          The spares do not work even though the part number in the database says it should. Then you look down and notice the part numbers are different. I'm looking at you IBM and your so-called equivalent FRU parts. Or worse all your spares are faulty and the last spare kills the entire unit. Now not only do you need a new PSU but new board :).

          Never ever say we have not had a failure in x time and do not expect one to ever happen.

          1. Will Godfrey Silver badge
            Unhappy

            Re: keeping swappable spares in stock might make sense

            The most fearsome phase on the packaging is:

            "XXXnn Compatible Replacement"

          2. MonkeyCee Silver badge

            Re: keeping swappable spares in stock might make sense

            You grab the box with the spare in it. The seal is broken...

      5. Anonymous Coward
        Anonymous Coward

        Re: keeping swappable spares in stock might make sense

        If my manager had a replacement for me on standby, I'd make sure to do a good job too!

        1. A.P. Veening Silver badge

          Re: keeping swappable spares in stock might make sense

          If my manager had a replacement for me on standby, I'd make sure to do a good job too!

          Seems to me it is more important to have a replacement for your manager standing by (but that goes for all managers).

      6. tekHedd

        Re: keeping swappable spares in stock might make sense

        "The mere act of having a spare in stock prevents it from breaking."

        According to Murphy's many corollaries, this only remains true as long as you are not depending on the presence of the spare to keep it from breaking.

      7. DougS Silver badge

        Re: keeping swappable spares in stock might make sense

        No. The mere act of having a spare in stock prevents it from breaking.

        I'm pretty sure this was proven to be part of quantum mechanics, similar to how observation collapses the wave function and a watched pot never boils.

    3. Doctor Syntax Silver badge

      Re: keeping swappable spares in stock might make sense

      "These are the costs that drive my manager batty."

      If he wants to he can always economise by not backing up data.

      1. chivo243 Silver badge
        Happy

        Re: keeping swappable spares in stock might make sense

        Not talking about data? Just the few extra power supplies that never got used. Yes, I understand the cost of doing business. So does my manager... hence the battiness.

    4. jmch Silver badge

      Re: keeping swappable spares in stock might make sense

      "When we replaced it, bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf"

      It's called insurance. Even if you never use the spare and end up junking it when the kit that it's supposed to cover is retired, it's still worth it. Ask the beancounters if they would ever skimp on, for example, building insurance. The case to be made is simple - what is the cost* of this item not working for a few minutes / days / hours / ever again. Then based on those numbers you devise an appropriate backup / redundancy strategy.

      * Although not all cost is strictly monetary, for beancounters everything can be reduced to (currency of choice) amount. Missed sales, man-hours needed to troubleshoot, emergence equipment sourcing / shipping cost... even intangibles such as customer confidence are in a company's balance sheet as 'Goodwill'. Sometimes you just need to know how to talk to beancounters in their own language

      1. imanidiot Silver badge

        Re: keeping swappable spares in stock might make sense

        The problem is that the decision NOT to buy spares is always made when the kit is first installed, spares are plentiful and the number of processes running on it are not too bloated, so cost of downtime isn't too great. So a beancounter looks at the numbers and veto's having spares because "the cost outweighs the benefit".

        Give it a few years and all those processes have grown in importance and a few got added. Meanwhile the manufacturer has come up with new shiny, shiny kit and the spares for your stuff now have to come from the "long term support center" at triple the cost and four times the delivery time. Downtime is now absolutely killing and guaranteed to be 4 times as long, but since the beancounters originally calculated you don't need spares, you can't have spares.

        1. Doctor Syntax Silver badge

          Re: keeping swappable spares in stock might make sense

          "Downtime is now absolutely killing and guaranteed to be 4 times as long, but since the beancounters originally calculated you don't need spares, you can't have spares."

          You then point out the revised cost/risk situation and that the only way to mitigate this is to replace the equipment now, before it fails and this time why not buy the spares.

        2. Anonymous Custard Silver badge
          Trollface

          Re: keeping swappable spares in stock might make sense

          and the spares for your stuff now have to come from the "long term support center"

          Or as the rest of us know it, eBay...

        3. jabuzz

          Re: keeping swappable spares in stock might make sense

          If it's IT kit and from a reputable manufacture, you head on over to the local tat bizzare that is eBay and buy the part as a used item from an IT recycler. The older method is to ring up a seller of refurbished parts and buy it from them direct though this tends to be more expensive in my experience.

          1. DougS Silver badge

            Re: keeping swappable spares in stock might make sense

            If you're buying refurbished parts you probably need to buy two so you WILL have a spare on hand, since that part refurbished by unknown means may not be as reliable as the already installed parts are.

      2. Fatman Silver badge
        Joke

        Re: keeping swappable spares in stock might make sense

        <quote>Sometimes you just need to know how to talk to beancounters in their own language</quote>

        I have always found that walking into a beancounter's office carrying a clue-by-four slung over my shoulder, gets their immediate attention.

        1. Charles 9 Silver badge
          FAIL

          Re: keeping swappable spares in stock might make sense

          Tried that. Found they were wearing hard hats because of "ongoing work in the room". Plus for some reason they had a security guard in the room, too.

    5. John 110
      Mushroom

      Re: keeping swappable spares in stock might make sense

      "...That said, we've not has a PS die in many a year. Luck?..."

      You really shouldn't have said that....

      1. Chris King Silver badge
        Mushroom

        Re: keeping swappable spares in stock might make sense

        "You really shouldn't have said that...."

        If I said the L-word, my place would go up like a Bond Baddie's hideout, hence the icon -->

    6. Joe W Silver badge

      Re: keeping swappable spares in stock might make sense

      Well, it is a gamble. Manglement needs to be reminded of 1) the stakes and 2) the probabilities, and 3) possible repercussions regarding bottom line / payouts / penalty payments - and preferably you need something in writing that states "yeah... nah, we won't buy replacement to set aside just in case"

    7. This post has been deleted by its author

    8. Alan Brown Silver badge

      Re: keeping swappable spares in stock might make sense

      "bought the spare PS, sat on the shelf, 6 years later, replaced that item, power supply still sits on the shelf."

      PSUs are cheap. Callouts are not.

      Anything containing a moving part should be considered a high risk potential point of failure

      I lost track of the number of times as a field tech I set out on on a job with "You only need XYZ" and found that I needed ABC JKH and NOP too. The days when I packed the wagon with EVERYTHING imaginable were the days when I found that someone had wired the power to the wrong circuit (non-essential power, etc) on the customer premises and my responsibility consisted of running a temporary extension lead so our batteries didn't go flat and telling them it needed to be fixed within 5 days, when I'd be back to checkup & sign off the fault or have them charged at full callout rates.

  6. Anonymous Coward
    Anonymous Coward

    Sounds like they were being a bit cheap to be honest...

    Big Cisco box, recessed dual power supplies? Capable of shifting 720Gbps around? Sounds like a Cat6500 with a SUP720 controller to me.

    If they'd gone with the SUP720-3C then they could have set up a pair of 6500s as a VSS pair to avoid this kind of thing - the old adage of "why have one when you can have two for twice the price".

    As a slight aside - does anyone know of a good way to remove redundant part numbers for old Cisco gear from using up space in my brain? I've tried beer but it hasn't worked so far...

    1. jmch Silver badge
      Pint

      Re: Sounds like they were being a bit cheap to be honest...

      " I've tried beer but it hasn't worked so far..."

      I think you haven't been trying hard enough.

      Have another ---->

      Still remember?

      Go to step 1

      1. A.P. Veening Silver badge

        Re: Sounds like they were being a bit cheap to be honest...

        Beer is just extremely immature whisky, I strongly recommend fully matured and ripened whisky, preferably cask strength.

        1. Charles 9 Silver badge

          Re: Sounds like they were being a bit cheap to be honest...

          Too risky. Ask Jim Beam. At least beer doesn't ignite.

          1. jake Silver badge
            Pint

            Re: Sounds like they were being a bit cheap to be honest...

            And with beer, you can enjoy a pint or two in the shower after a long afternoon of mowing. Whisky, probably not so much.

            About to test my theory ... not with the whiskey, I'll leave that as an exercise for the reader ...

            1. A.P. Veening Silver badge

              Re: Sounds like they were being a bit cheap to be honest...

              And with beer, you can enjoy a pint or two in the shower

              Beer already resembles making love in a canoe too much and now you are making it even worse by diluting it.

              1. Charles 9 Silver badge

                Re: Sounds like they were being a bit cheap to be honest...

                Ever thought that's EXACTLY WHY people enjoy beer? It's as close to water without actually BEING water, AND it still provides a nice buzz (some people don't really want to get drunk--those that do--usually out of desperation or deep depression--resort to cheap liquors and bum wines). Always consider the environment: a lot of Americans are experiencing 40-degree weather right now and could really stand a cold one, not JUST for the buzz.

    2. Chris King Silver badge

      Re: Sounds like they were being a bit cheap to be honest...

      If they'd gone with the SUP720-3C then they could have set up a pair of 6500s as a VSS pair to avoid this kind of thing - the old adage of "why have one when you can have two for twice the price".

      Maybe the Pork Barrel wasn't quite so deep that year ?

  7. Rich 11 Silver badge

    Ever had an "oops, butterfingers!" moment and taken out a government agency?

    Well, there was that time I accidentally dropped Chris Grayling into the Ministry of Justice, and before I knew it two-thirds of the Probation Service had vanished.

    1. TheFirstChoice

      Re: Ever had an "oops, butterfingers!" moment and taken out a government agency?

      ...and he was then fumbled into the Department for Transport and two-thirds of the train services disappeared?

      1. Down not across Silver badge

        Re: Ever had an "oops, butterfingers!" moment and taken out a government agency?

        ...never mind. He got some ferries instead. Oh wait...

  8. Anonymous Coward
    Anonymous Coward

    Oh the joys of on site spares...

    It's gets better when the equipment is over spec'd enough that it lasts for years... And years and the company that made said kit gets bought out. And changes its name.... Twice (as does the company that uses the kit).

    Before finally a battery dies in the system and no one notices. Or has a spare. Because the equipment is 15 years old (and 10+ years since it was last DR tested).

    Yeah. Been there before (battery was a raid array battery and because it wasn't my own kit I couldn't juryrig some batteries to get it working with a soldering iron and persistence). Spare part was located in a different city and shipped over same day and fitted within 30 mins of going to fail over systems.

    Also happened to be part of a billing system for a very large teleco. Fun times.

    Anon because I'm still working for them and I'm pushing for replacement of the kit should it die again (once bitten, twice shy)

  9. Sequin

    I worked on a site where the electricians had the wonderful idea of installing an RCD circuit breaker between the UPS and the Server - workers in the road outside accidentally cut the incoming mains supply and the RCD tripped, isolating the server from the power supply.

    1. Anonymous Coward
      Anonymous Coward

      Electricians

      Important Rule..... never let electricians near any electricity they don't understand! Including telecoms, networks etc.

      Anon, as I know lots of electricians...

      1. Benson's Cycle

        Re: Electricians

        If you didn't allow electricians near any electricity they don't understand, not much work would get done.

        Though I do know an exception...he has a degree in English and he actually understands the wiring regs.

        1. Alan Brown Silver badge

          Re: Electricians

          The trick is to explain what you want done in small words, in writing (so they can't argue about what was said), have a copy of the regs to hand where they can see you have it and most importantly, make sure they understand that if they don't do it to a working, certifiable and satisfactory standard, they don't get paid.

          This last bit is crucial to getting their complete undivided attention AND have them ask questions if they think the task is odd, instead of just bodging it.

          For added attention you can add that if someone else has to fix it, you'll bill them for the rework.

  10. Anonymous Coward
    Anonymous Coward

    Half-redundant feeds

    Reminds me of a situation I once encountered with a piece of network kit that had two (redundant) PSUs. Each PSU had a small handle at each end to allow it to be smoothly pulled out of the chassis.

    The problem was that whoever had installed them had used the handles on PSU B as cable management for the power feed to PSU A. There was insufficient slack in the cables so, if PSU B failed, it was impossible to replace without first disconnecting PSU A.

    1. Alistair Silver badge

      Re: Half-redundant feeds

      A certain corporate entity delivers 4 prepack racks of proprietary exalent (no, not a spelling error) servers. 2 10Ge fibre cables, 2 Infiniband cables, 2 10Ge ethernet cables, 1 100Mb console port cable, 2 power cables. Fitted to 1U servers, 36 to a rack.

      Issue: replace one of the 1U infiniband switches on the *bottom* of the rack, WITHOUT taking out the entire rack of servers. Go for it. anyone? Bueler?

  11. Claverhouse Silver badge
    Happy

    Taking Life the Hard Way

    Representatives of the military "showed up the next day and ripped out their circuit.

    "They weren't smiling."

    I dunno what it is about modern militaries; all seem fairly humourless. Prolly more efficient than in the past, but as far as I can see --- unless they were a naturally surly bastard of an NCO, most such emissaries in the British Armed Forces back in 1925, 1945, or 1965 would have been fairly sanguine and said things like, "Well, there you go..." and "Worse things happen at sea...". Grimness was left to the tedious Ivans...

  12. Androgynous Cupboard Silver badge

    On the plus side

    Somewhere in Northern Pakistan, a wedding went off without a hitch and everyone went home alive.

    No? Too soon?

  13. Anonymous Coward
    Anonymous Coward

    Soon... soon....

    It's too soon to tell my own 'Who, me?' story. Give me another year or two. Or three.

    A/C (for blindingly obvious reasons).

    1. It's just me

      Re: Soon... soon....

      Just waiting for the statute of limitations to expire, huh?

  14. jtaylor

    My favorite redundant power supply

    was on the Sun E450. https://docs.oracle.com/cd/E19088-01/450.srvr/805-0429-10/805-0429-10.pdf (page 84)

    3 power supplies, redundant and hot swappable.

    1 power cord.

    Someone at Sun was taking the piss.

    1. Anonymous Coward
      Anonymous Coward

      Re: My favorite redundant power supply

      Separate power sources protect against a different set of failures from separate power supplies. In this case, the E450 was protected against single PSU failure. If you also wanted to protect against the failure of a single phase, transformer, UPS, etc. then you would also need to use 3 separate power cords and 3 independent sources. The fact that most of the time we want both does not mean there's no value in the one.

      Many hardware designers operate under the naive belief that moving parts are the most common source of failure, so they provide redundancy in anything that has a fan while usually ignoring everything else. In reality the most common sources of failure are operator error and firmware/software defect, in that order, so most of the time that's not enough.

      1. Alan Brown Silver badge

        Re: My favorite redundant power supply

        "In reality the most common sources of failure are operator error and firmware/software defect, in that order, so most of the time that's not enough."

        Fans come in third though, and most of them aren't field-replaceable items (Says me, who's had to resort to changing bearing races in PSU fans for one model of disk array that Xyratex stopped making 7 years ago, thanks to budgetary constraints making it impossible to replace the arrays - yet)

  15. Anonymous Coward
    Anonymous Coward

    HSRP (Holy Sh*te Really Pathetic)

    No network with that kind of throughput isn't without multiple paths to get layer 3 from one place to another. If the phone call wasn't to ask why the standby router/path had taken over, I'd be asking why such a high throughput system wasn't asking me that question.

  16. cyberdemon
    Coat

    From the title, I thought this was going to be a shock story about ROHS compliance

    But instead it was about swinging the lead

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019