back to article Human error blamed for toxic Russian rocket explosion

Russia's latest rocket mishap, which saw three of the Glonass satellites needed to upgrade the country's global positioning system destroyed in a fiery inferno, is reported to have been the fault of a technician not knowing which way is up. After sorting through the wreckage, the investigating team has found a flight control …

COMMENTS

This topic is closed for new posts.
  1. Brian Miller
    Meh

    Nothing new

    Different day, different country, same idiots. When I was in the Army, a communications van was set on fire when its power wires were hooked up wrong. (Yes, it was all color coded. Couldn't get it wrong, no sir...)

    1. Wzrd1 Silver badge

      Re: Nothing new

      Look up the origin of Murphy's law. It was literally a case of a end about wiring job on a sensor that made Captain Murphy make a remark so similar to the one we so enjoy today.

      Though, it remains suspect if the technician was at fault or the engineer who built the sensor...

      1. Anonymous Coward
        Anonymous Coward

        technician or the engineer at fault?

        Engineer.

        What sort of idiot makes a uni-direction part that can be installed in several orientations?

        What sort of idiot can't follow a simple arrow?

        What sort of idiot makes a guess when installing parts in a rocket?

  2. Anonymous Coward
    Anonymous Coward

    Typical Management.

    Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?

    This is management error, but let's not even say error. This was the result of management's decision to gamble.

    1. cosymart
      Coat

      Re: Typical Management.

      It had an arrow on it. What more do you want :-) I suppose it all depends on the orientation of the bit of kit when it was assembled.....

      The coat with the natty black arrows design please.

      1. Wzrd1 Silver badge

        Re: Typical Management.

        From what I saw, rockets are assembled on their sides in stages/levels of assembly. No up or down, unless one is pointing up then, which would be to the side.

        Even so, what did the engineering plans show the technician on how to install that sensor? I remember many decades ago seeing power supply after power supply (OK, two before I halted smoke testing) produce a mushroom cloud of smoke.

        Turns out that the installation diagrams for the installation techs shorted the output of the rectifiers, sending 100A into a dead short on one pole of the filter capacitors, rather than each lead going to each lead of the filter capacitor bank.

        Then, change management wanted to hold a committee meeting about it. That screwed up three days of production.

    2. John Smith 19 Gold badge
      Unhappy

      Re: Typical Management.

      "Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?

      This is management error, but let's not even say error. This was the result of management's decision to gamble."

      According to the JPL "lessons Learned" database power supply problems (including wrong voltage, wrong current, wrong polarity, wrong power up sequence) are the #1 cause of orbital payloads being either degraded or destroyed.

      And this is in an environment where everyone is aware of the consequences of failure (into the $Bn) and many of them can't be fixed.

      1. Yet Another Anonymous coward Silver badge

        Re: Typical Management.

        > Did the test plan check for the error?

        It did - the Russian test plan is generally to fire the thing and see if it blows up.

        For bonus points make sure you test it near Germans - in which case it doesn't matter if it worked or failed.

    3. Cliff

      Re: Typical Management.

      Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago.

      1. Muckminded

        Re: Typical Management.

        Maybe if management were part of the payload, the work would improve.

      2. Wzrd1 Silver badge

        Re: Typical Management.

        "Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."

        Management issue to ensure design is up to date with Captain Murphy's law.

        It's only annoying that it took so long for undergrads to learn about Murphy's law when considering design.

        Especially since Captain Murphy served back in the late 1940's and the actual term has been dated to 1877 at the current earliest found reference from one Alfred Holt.

        That said, I'd not be surprised to learn that the reality of it is, such was first uttered in the early days of complex lever/pivot systems or complex pulley systems were first employed.

      3. Charles 9

        Re: Typical Management.

        Except you forget the adage about the foolproof: never underestimate the ingenuity of complete fools. Even if you design it with one-way connectors, some complete buffoon who managed to slip through the cracks will find SOME way to jam it in the wrong way.

      4. BristolBachelor Gold badge

        @ Cliff

        "Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."
        That is fine, but normally wires come in standard sizes, and the article says that a part was wired up with incorrect polarity. If someone swaps two wires in a connector or on a component, then it is the fault of the wireman, and later the inspector, and tester (possibly a design error if the designer has not allowed a mechanism to test for this failure.)

        1. This post has been deleted by its author

        2. Solmyr ibn Wali Barad

          Re: @ Cliff

          "If someone swaps two wires in a connector or on a component"

          This.

          Unidirectional connectors can reduce human errors, but not eliminate them. Did we really think that Murphy is so easy to defeat?

          Next suggestion: Use a machine! Now we're immune against human errors and Murphy's laws. He he.

  3. cosymart
    Mushroom

    Big Bang

    One of the more impressive failures, complete with blasts, bangs and (assumed) Russian expletives.

    1. Alan Esworthy
      Pint

      Re: Big Bang Expletives

      По большой голубой Санкт Кирилла яички, кто-то за это заплатит!

      1. bazza Silver badge

        Re: Big Bang Expletives

        "By St. Cyril of great blue testicles, someone will pay for it!"

        Or is Google Translate being overly modest?!

        1. Mike 137 Silver badge

          Re: Big Bang Expletives

          Strictly "by the big blue egg [singular, familiar form] (or testicle [singular]) of St. Cyril..."

        2. Turtle

          Re: Big Bang Expletives

          "Or is Google Translate being overly modest?"

          No, that's very mild. No real obscenities in the cited Russian sentence. Just a bit of euphemism.

          (Note that the following is an unnecessary excursus on colloquial Russian. There's no need for anyone to read it...)

          The Russian language has a very extensive collection of expletives, known as "mat' ". (I'm not going to bother typing in Russian here, so forgive me.) See for example Flegon's "Za Predelami Russkix Slovarei" which, to be sure, includes not only "real" mat' but also euphemistic cases, such as the aforementioned "yaichka" (a diminutive form of the feminine noun "yaitsa" in the nominative singular) which translates, literally, as "little egg" but which here means, well, you know. Dictionaries of "blatnaya muzyka" (thieves' cant) and prison-camp slang are also very heavily larded with the various uses of profanity and obscenity, much of which is due to the way that Russian forms words by the use of prefixes, suffixes, and diminutives.

          Whereas, for example, one would take a verb, such as " et' " (to fuck, either literally or figuratively) and by combining it with the prefix "s' " (meaning, among other things, "motion away from") and putting a verb ending followed by a reflexive particle "(vats'ya" ) you would end up with one word, s'ebyvat'sya" which is an exactly equivalent of the the English "fuck off". I.e. Russian forms new words, where English collocates existing words into set expressions. (I'm simplifying here, of course, but it's generally correct.) This method of word formation has enabled the very, very luxurious growth of Russian obscenities into an extremely extensive and expressive lexicon.

          (Note that the stray apostrophes here are for two Russian "letters" which do not in themselves signify any sounds, but indicate whether the consonant preceding them is palatized (palatalized) or not. Well, don't ask. Really.)

  4. John Smith 19 Gold badge
    Unhappy

    That's not a good track record to have.

    Maybe time to consider retiring this design?

    Just a thought.

    1. Wzrd1 Silver badge

      Re: That's not a good track record to have.

      Hmm, let's consider things a bit.

      The US was notorious, to the great mirth of the nations of the world, for having rockets that went in reverse or simply fell over and exploded.

      When we finally managed to get them to go up, they frequently disassembled themselves or decided to take a random, if toward the launch/recording bunker is random direction and be destructed (one film that I saw had the range safety officer thanking the officer that destructed the defective missile, said officer's response, I didn't, destruct failed).

      Much, much, much later, the Apollo program began. The rockets had major pogo oscillation problems.

      By your candle, the rocket would've been retired.

      Nope, it was partially redesigned to remove the flex that caused the interruption in fuel/oxidant flow. From that fruit, a bunch of men managed to not walk on the moon, but hop, jump, shuffle and most often, fall on their asses or knees, largely asses.

      NASA hit that one from the world for a long time. Pity, it's funny and educational. Low gravity equals grave risk of suit rupture.

      Thankfully, the suits were overdesigned to compensate for the unknown.

      Which resulted in a significant number of said spills.

      1. AndyS

        Re: That's not a good track record to have.

        Wzrd1, what are you wittering on about?

  5. Mr Young
    Thumb Up

    Give the poor guy a break!

    Looks like the rocket performed really well considering some sensor was upside down;

    1. Yet Another Anonymous coward Silver badge

      Re: Give the poor guy a break!

      Simply change the marketing material to "new automated mining system in trial of Russia - Australia direct tunnel" and you have a success.

  6. fnusnu

    Not for the first time...

    https://en.wikipedia.org/wiki/Sutton_Wick_air_crash

  7. Don Jefe

    Directional Arrows

    Directional arrows alone are hardly ever enough for complex systems. There are too may ways to obscure the markings and there's always the real chance that 'up' doesn't mean up when the system is in the ready state. If a component is directional it is also important and adding physical deterrents to prevent this from happening generally doesn't add too much cost considering most of the parts are custom or semi-custom anyway.

    Ideally complex systems meant for production should mostly snap together so there is no room for error and assembly doesn't require specialists with advanced degrees. A day or two of training should have anyone ready to start assembling rockets. We know it can be done, WWII proved that COTS Humans could do this stuff if it was designed well. At the end of the day it is all about cost savings, bureaucracy sucks up so much of a projects funds there isn't enough left to properly engineer the project.

    1. Yet Another Anonymous coward Silver badge

      Re: Directional Arrows

      Or if you are designing a component where orientation matters - don't put a symmetrical set of mounting holes.

      1. Vladimir Plouzhnikov

        Re: Directional Arrows

        According to Russian press it was not orientation but swapped polarity.

        Regarding asymmetrical mountings etc - for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof. Maybe it's laziness, maybe arrogance, but most likely the consequence of the "planned" economy - all small things were supposed to be "universal", "standardised" and produced by the millions. If your rocket needs a non-standard connector - you're doing it 'rong...

        1. Wzrd1 Silver badge

          Re: Directional Arrows

          "...for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof."

          Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression.

          Still, when the full failure analysis is completed, there will be the public response and the quiet response to idiot proof systems, for even the brightest person on the planet can have his or her village idiot moment, something that space programs won't tolerate due to the massive failures involved.

          It's not prestige, it's finance. Failures cost, both financially and in client base.

          And the Russians are no fools.

          And as my father frequently said over the years, "My father may have raised a dummy, but he didn't raise a fool".

          EVERY nation knows that as a truism, though the definition of dummy is variable based up speaker.

          1. TeeCee Gold badge
            Happy

            Re: Directional Arrows

            "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools."

            Ford Prefect.

          2. Turtle

            @Wzrd1 Re: Directional Arrows

            "Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression."

            Not strictly apposite to your comment but the phrase "There's never enough time to do it right but there's always enough time to do it over" did come to mind.

            (I guess we'd have to change that to "There's never enough money to do it right but there's always enough to do it over.")

      2. FuzzyTheBear
        Pint

        Re: Directional Arrows

        ... and you will find a diligent fix it artist with a drill and a tap working a new set of holes because they are in the wrong direction. " I'll fix it , don't worry ... "

    2. Wzrd1 Silver badge

      Re: Directional Arrows

      "Ideally complex systems meant for production should..."

      The problem is, space flight isn't an ideal situation. Mass is of grave concern. Do the snap in approach, add in hundreds of pounds of additional mass at best, a ton or so (tonne for some out there) at a more realistic approach.

      That said, whereinhell was QC who should inspect critical assemblies? Of which, nearly every assembly is critical in a rocket designed to go into orbit or near orbit.

      For the price per unit mass, I'd not have one QC inspection, but three, considering the Russian labor rate.

      1. monkeyfish

        Re: Directional Arrows

        Yeah, but you know what happens when 3 people check something independently?

        Person 1: I've looked at it a bit, that'll do, there's 2 other people checking this after me.

        Person 2: I'm sure person 1 checked it, and there's always person 3, so I'm knocking off early.

        Person 3: Persons 1 & 2 already checked it, so what's the point of doing it again?

        1. PC Paul

          Re: Directional Arrows

          Apply game theory: If a QC inspector finds a mistake one of the others didn't he gets a big bonus. If he finds a mistake both the others missed, he gets a MASSIVE bonus.

          If a QC inspector misses a mistake found by one other, he pays a penalty. If he misses a mistake found by two others (and hence easier to find), he goes to Siberia.

          NOW they will focus on the job...

          Although I recall a tale of a company paying testers per bug found and paying coders per bug fixed. Unfortunately the coders and the testers spoke to each other and it all got quite lucrative...

  8. Anonymous Coward
    Anonymous Coward

    easy mistake to make ..

    I expect they simply mistook a metric 'up' for an imperial 'up' .....

  9. Anonymous Coward
    Anonymous Coward

    What troubles me is that for the first 17 seconds the rocket must have thought it was flying DOWN.

    1. Wzrd1 Silver badge

      "What troubles me is that for the first 17 seconds the rocket must have thought it was flying DOWN."

      What fascinates me is that for the first 17 seconds, the rocket sensed it was flying down, but operated correctly.

      That, in and of itself is amazing. It suggests a software solution to the issue.

      Some only see a problem, others see the exception and find a solution based upon that exception to capitalize upon.

      1. Vladimir Plouzhnikov

        Again, the Russian sources say that the problem was an "angular velocity" sensor, don't know in which coordinate.

        From the footage of the launch, the rocket looks spinning quite fast, so probably that was the roll rate sensor.

        That would perfectly explain why the lift off looked normal for a few seconds and then the system went into positive feedback, trying to correct a slight roll but instead speeding it up.

  10. Cubical Drone

    I get what you are all saying about design, but...

    ...there has to come a point when you also need to hire competent people.

    In a prefect world you could design something that is fool proof, but you should always remember, "if it is something any idiot can do, only an idiot will do it."

  11. John Pattenden
    Boffin

    up<--->dn

    Infallible marking system.

    1. Don Jefe

      Re: up<--->dn

      It is only infallible if the use state of the 'thing' corresponds with the alignment markings on the part. Or, up during assembly may not be up when in flight.

    2. John Pattenden
      Joke

      Re: up<--->dn

      Oh dear. I forgot my own infallible marking system...

      <sarc>

      Infallible marking system

      </sarc>

      <hint>

      Try reading the title upside down.

      </hint>

    3. Cliff

      Re: up<--->dn

      Clever, I like it :)

      Took me a moment and your hint, but that was worth it

  12. Jon Green

    What about pre-flight tests?

    Any sensible in-flight systems (not to mention remote telemetry setups) ought to have picked up on anomalous sensor readings and aborted before the motors were even ignited.

    Sounds like failures all along the line, from sloppy design, through sloppy assembly.

    1. Malcolm Weir Silver badge
      FAIL

      Re: What about pre-flight tests?

      Err.. the sensor was apparently a vertical motion sensor. So until the thing actually starts to move, no-one knows there's a problem.

      1. Chris 244
        Flame

        Re: Er, Fail Comment Fail

        If the only way to test your equipment is to press the big red button ...

        ... I'll just wait to catch the next flight, thank you.

      2. Jon Green
        FAIL

        Re: What about pre-flight tests?

        ...and if they've got a brain cell, there will be multiple redundancy. So if one sensor says "down" and all the others say "up", the on-board computer marks the anomalous sensor defective and ignores it unless or until it reliably returns to service or the OBC's telecommanded to bring it back into the fold.

        Or, as in this case, not.

        I write on-board software for space systems, so I do have a little insight into this.

        1. Charles 9

          Re: What about pre-flight tests?

          Redundancy adds weight, and rockets are VERY weight-sensitive, meaning you run the risk of it being too heavy to fly.

  13. mika7367
    FAIL

    Techie Problem

    The Russians are welcome to have one of my spare Linux boxes, I'll even go over and run maintenance on it once a month :)

    1. Crazy Operations Guy

      Re: Techie Problem

      How is that going to fix anything?

    2. Wzrd1 Silver badge

      Re: Techie Problem

      Only so that you can see the next potential fireworks, right?

  14. Anonymous Coward
    Anonymous Coward

    Don't make me turn this rocket around

    The three kids (Glonass) were flighting in the back.

  15. Paul Hovnanian Silver badge
    Joke

    Ah yes.

    The Russian reversal in another form.

  16. An(other) Droid
    Mushroom

    Amazing!

    The connector survived this? ----------------------------------------->

    And with legible markings!?

    I would have thought every would have burnt to ashes, if not vaporized.

    Well, how very convenient for the investigators ;-)

    1. Vulch
      Boffin

      Re: Amazing!

      One advantage of hypergolic fuels is that they self ignite as soon as they mix so you don't get much of an explosion. The American Titan II used the same sort of fuel which led to the Gemini launch escape system being ejector seats instead of the solid rockets used on Mercury and Apollo.

  17. Geoff Campbell Silver badge
    Boffin

    Does this mean we've been lied to all these years?

    I thought reversing the polarity was supposed to cure all problems?

    GJC

  18. ratfox
    Happy

    Easy to fix

    Just reverse the polarity of the Proton beams…

  19. The Serpent

    No no no. You reverse the polarity of the neutron flow

  20. tomban
    Alien

    Human error blamed...

    We /know/ it was human error, duh, but what error did the humans do?

  21. Irongut

    As my Kerbals will tell you this rocket surgery thing is real tricky, especially the flying part. Up, down, left, right... they're all relative and when you get into a spin you could end up going in any direction.

  22. Ed 13

    Ariane 5

    That's all I have to say.

  23. cortland

    You are under arrest

    UP is a state secret.

  24. Apollo-Soyuz 1975

    I hope I'm not the only one wondering why there wasn't a range safety system to blow the rocket up _before_ it hit the ground...

This topic is closed for new posts.

Other stories you might like