Feeds

back to article Human error blamed for toxic Russian rocket explosion

Russia's latest rocket mishap, which saw three of the Glonass satellites needed to upgrade the country's global positioning system destroyed in a fiery inferno, is reported to have been the fault of a technician not knowing which way is up. After sorting through the wreckage, the investigating team has found a flight control …

COMMENTS

This topic is closed for new posts.

Page:

Bronze badge
Meh

Nothing new

Different day, different country, same idiots. When I was in the Army, a communications van was set on fire when its power wires were hooked up wrong. (Yes, it was all color coded. Couldn't get it wrong, no sir...)

1
0
Bronze badge

Re: Nothing new

Look up the origin of Murphy's law. It was literally a case of a end about wiring job on a sensor that made Captain Murphy make a remark so similar to the one we so enjoy today.

Though, it remains suspect if the technician was at fault or the engineer who built the sensor...

2
0
Anonymous Coward

technician or the engineer at fault?

Engineer.

What sort of idiot makes a uni-direction part that can be installed in several orientations?

What sort of idiot can't follow a simple arrow?

What sort of idiot makes a guess when installing parts in a rocket?

0
0
Anonymous Coward

Typical Management.

Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?

This is management error, but let's not even say error. This was the result of management's decision to gamble.

5
0
Coat

Re: Typical Management.

It had an arrow on it. What more do you want :-) I suppose it all depends on the orientation of the bit of kit when it was assembled.....

The coat with the natty black arrows design please.

2
0
Gold badge
Unhappy

Re: Typical Management.

"Why did no one catch the error? Did the test plan check for the error? Did the design allow for testing for the error? Why did the design not prevent the error with, say, an asymmetrical connector?

This is management error, but let's not even say error. This was the result of management's decision to gamble."

According to the JPL "lessons Learned" database power supply problems (including wrong voltage, wrong current, wrong polarity, wrong power up sequence) are the #1 cause of orbital payloads being either degraded or destroyed.

And this is in an environment where everyone is aware of the consequences of failure (into the $Bn) and many of them can't be fixed.

2
0
Silver badge

Re: Typical Management.

> Did the test plan check for the error?

It did - the Russian test plan is generally to fire the thing and see if it blows up.

For bonus points make sure you test it near Germans - in which case it doesn't matter if it worked or failed.

5
0
Silver badge

Re: Typical Management.

Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago.

19
0

Re: Typical Management.

Maybe if management were part of the payload, the work would improve.

2
0
Bronze badge

Re: Typical Management.

From what I saw, rockets are assembled on their sides in stages/levels of assembly. No up or down, unless one is pointing up then, which would be to the side.

Even so, what did the engineering plans show the technician on how to install that sensor? I remember many decades ago seeing power supply after power supply (OK, two before I halted smoke testing) produce a mushroom cloud of smoke.

Turns out that the installation diagrams for the installation techs shorted the output of the rectifiers, sending 100A into a dead short on one pole of the filter capacitors, rather than each lead going to each lead of the filter capacitor bank.

Then, change management wanted to hold a committee meeting about it. That screwed up three days of production.

3
0
Bronze badge

Re: Typical Management.

"Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."

Management issue to ensure design is up to date with Captain Murphy's law.

It's only annoying that it took so long for undergrads to learn about Murphy's law when considering design.

Especially since Captain Murphy served back in the late 1940's and the actual term has been dated to 1877 at the current earliest found reference from one Alfred Holt.

That said, I'd not be surprised to learn that the reality of it is, such was first uttered in the early days of complex lever/pivot systems or complex pulley systems were first employed.

0
0
Silver badge

Re: Typical Management.

Except you forget the adage about the foolproof: never underestimate the ingenuity of complete fools. Even if you design it with one-way connectors, some complete buffoon who managed to slip through the cracks will find SOME way to jam it in the wrong way.

1
0
Gold badge

@ Cliff

"Design issue - if something is only meant to fit in one orientation, you design so it can only fit in one orientation. This was undergrad stuff 20+ years ago."
That is fine, but normally wires come in standard sizes, and the article says that a part was wired up with incorrect polarity. If someone swaps two wires in a connector or on a component, then it is the fault of the wireman, and later the inspector, and tester (possibly a design error if the designer has not allowed a mechanism to test for this failure.)

0
0

This post has been deleted by its author

Bronze badge

Re: @ Cliff

"If someone swaps two wires in a connector or on a component"

This.

Unidirectional connectors can reduce human errors, but not eliminate them. Did we really think that Murphy is so easy to defeat?

Next suggestion: Use a machine! Now we're immune against human errors and Murphy's laws. He he.

0
0
Mushroom

Big Bang

One of the more impressive failures, complete with blasts, bangs and (assumed) Russian expletives.

2
0
Pint

Re: Big Bang Expletives

По большой голубой Санкт Кирилла яички, кто-то за это заплатит!

5
0
Silver badge

Re: Big Bang Expletives

"By St. Cyril of great blue testicles, someone will pay for it!"

Or is Google Translate being overly modest?!

2
0

Re: Big Bang Expletives

Strictly "by the big blue egg [singular, familiar form] (or testicle [singular]) of St. Cyril..."

0
0
Silver badge

Re: Big Bang Expletives

"Or is Google Translate being overly modest?"

No, that's very mild. No real obscenities in the cited Russian sentence. Just a bit of euphemism.

(Note that the following is an unnecessary excursus on colloquial Russian. There's no need for anyone to read it...)

The Russian language has a very extensive collection of expletives, known as "mat' ". (I'm not going to bother typing in Russian here, so forgive me.) See for example Flegon's "Za Predelami Russkix Slovarei" which, to be sure, includes not only "real" mat' but also euphemistic cases, such as the aforementioned "yaichka" (a diminutive form of the feminine noun "yaitsa" in the nominative singular) which translates, literally, as "little egg" but which here means, well, you know. Dictionaries of "blatnaya muzyka" (thieves' cant) and prison-camp slang are also very heavily larded with the various uses of profanity and obscenity, much of which is due to the way that Russian forms words by the use of prefixes, suffixes, and diminutives.

Whereas, for example, one would take a verb, such as " et' " (to fuck, either literally or figuratively) and by combining it with the prefix "s' " (meaning, among other things, "motion away from") and putting a verb ending followed by a reflexive particle "(vats'ya" ) you would end up with one word, s'ebyvat'sya" which is an exactly equivalent of the the English "fuck off". I.e. Russian forms new words, where English collocates existing words into set expressions. (I'm simplifying here, of course, but it's generally correct.) This method of word formation has enabled the very, very luxurious growth of Russian obscenities into an extremely extensive and expressive lexicon.

(Note that the stray apostrophes here are for two Russian "letters" which do not in themselves signify any sounds, but indicate whether the consonant preceding them is palatized (palatalized) or not. Well, don't ask. Really.)

5
0
Gold badge
Unhappy

That's not a good track record to have.

Maybe time to consider retiring this design?

Just a thought.

0
0
Bronze badge

Re: That's not a good track record to have.

Hmm, let's consider things a bit.

The US was notorious, to the great mirth of the nations of the world, for having rockets that went in reverse or simply fell over and exploded.

When we finally managed to get them to go up, they frequently disassembled themselves or decided to take a random, if toward the launch/recording bunker is random direction and be destructed (one film that I saw had the range safety officer thanking the officer that destructed the defective missile, said officer's response, I didn't, destruct failed).

Much, much, much later, the Apollo program began. The rockets had major pogo oscillation problems.

By your candle, the rocket would've been retired.

Nope, it was partially redesigned to remove the flex that caused the interruption in fuel/oxidant flow. From that fruit, a bunch of men managed to not walk on the moon, but hop, jump, shuffle and most often, fall on their asses or knees, largely asses.

NASA hit that one from the world for a long time. Pity, it's funny and educational. Low gravity equals grave risk of suit rupture.

Thankfully, the suits were overdesigned to compensate for the unknown.

Which resulted in a significant number of said spills.

0
1

Re: That's not a good track record to have.

Wzrd1, what are you wittering on about?

0
0
Thumb Up

Give the poor guy a break!

Looks like the rocket performed really well considering some sensor was upside down;

2
0
Silver badge

Re: Give the poor guy a break!

Simply change the marketing material to "new automated mining system in trial of Russia - Australia direct tunnel" and you have a success.

2
0

Not for the first time...

https://en.wikipedia.org/wiki/Sutton_Wick_air_crash

0
0
Silver badge

Directional Arrows

Directional arrows alone are hardly ever enough for complex systems. There are too may ways to obscure the markings and there's always the real chance that 'up' doesn't mean up when the system is in the ready state. If a component is directional it is also important and adding physical deterrents to prevent this from happening generally doesn't add too much cost considering most of the parts are custom or semi-custom anyway.

Ideally complex systems meant for production should mostly snap together so there is no room for error and assembly doesn't require specialists with advanced degrees. A day or two of training should have anyone ready to start assembling rockets. We know it can be done, WWII proved that COTS Humans could do this stuff if it was designed well. At the end of the day it is all about cost savings, bureaucracy sucks up so much of a projects funds there isn't enough left to properly engineer the project.

2
0
Silver badge

Re: Directional Arrows

Or if you are designing a component where orientation matters - don't put a symmetrical set of mounting holes.

1
0
Silver badge

Re: Directional Arrows

According to Russian press it was not orientation but swapped polarity.

Regarding asymmetrical mountings etc - for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof. Maybe it's laziness, maybe arrogance, but most likely the consequence of the "planned" economy - all small things were supposed to be "universal", "standardised" and produced by the millions. If your rocket needs a non-standard connector - you're doing it 'rong...

1
0
Bronze badge

Re: Directional Arrows

"Ideally complex systems meant for production should..."

The problem is, space flight isn't an ideal situation. Mass is of grave concern. Do the snap in approach, add in hundreds of pounds of additional mass at best, a ton or so (tonne for some out there) at a more realistic approach.

That said, whereinhell was QC who should inspect critical assemblies? Of which, nearly every assembly is critical in a rocket designed to go into orbit or near orbit.

For the price per unit mass, I'd not have one QC inspection, but three, considering the Russian labor rate.

1
0
Bronze badge

Re: Directional Arrows

"...for whatever reason the Soviet, now Russian, industry never bothered to make the little things foolproof."

Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression.

Still, when the full failure analysis is completed, there will be the public response and the quiet response to idiot proof systems, for even the brightest person on the planet can have his or her village idiot moment, something that space programs won't tolerate due to the massive failures involved.

It's not prestige, it's finance. Failures cost, both financially and in client base.

And the Russians are no fools.

And as my father frequently said over the years, "My father may have raised a dummy, but he didn't raise a fool".

EVERY nation knows that as a truism, though the definition of dummy is variable based up speaker.

0
0
Gold badge
Happy

Re: Directional Arrows

"A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools."

Ford Prefect.

0
0
Silver badge

@Wzrd1 Re: Directional Arrows

"Their problem was that of budget. A serious budget, with little to no cost overrun tolerance. It wasn't unique to the space program, it was endemic to anything not military during the Cold War, later to everything due to economic depression."

Not strictly apposite to your comment but the phrase "There's never enough time to do it right but there's always enough time to do it over" did come to mind.

(I guess we'd have to change that to "There's never enough money to do it right but there's always enough to do it over.")

0
0
Bronze badge

Re: Directional Arrows

Yeah, but you know what happens when 3 people check something independently?

Person 1: I've looked at it a bit, that'll do, there's 2 other people checking this after me.

Person 2: I'm sure person 1 checked it, and there's always person 3, so I'm knocking off early.

Person 3: Persons 1 & 2 already checked it, so what's the point of doing it again?

4
0

Re: Directional Arrows

Apply game theory: If a QC inspector finds a mistake one of the others didn't he gets a big bonus. If he finds a mistake both the others missed, he gets a MASSIVE bonus.

If a QC inspector misses a mistake found by one other, he pays a penalty. If he misses a mistake found by two others (and hence easier to find), he goes to Siberia.

NOW they will focus on the job...

Although I recall a tale of a company paying testers per bug found and paying coders per bug fixed. Unfortunately the coders and the testers spoke to each other and it all got quite lucrative...

1
0
Pint

Re: Directional Arrows

... and you will find a diligent fix it artist with a drill and a tap working a new set of holes because they are in the wrong direction. " I'll fix it , don't worry ... "

1
0
Anonymous Coward

easy mistake to make ..

I expect they simply mistook a metric 'up' for an imperial 'up' .....

12
0

What troubles me is that for the first 17 seconds the rocket must have thought it was flying DOWN.

2
0
Bronze badge

"What troubles me is that for the first 17 seconds the rocket must have thought it was flying DOWN."

What fascinates me is that for the first 17 seconds, the rocket sensed it was flying down, but operated correctly.

That, in and of itself is amazing. It suggests a software solution to the issue.

Some only see a problem, others see the exception and find a solution based upon that exception to capitalize upon.

0
0
Silver badge

Again, the Russian sources say that the problem was an "angular velocity" sensor, don't know in which coordinate.

From the footage of the launch, the rocket looks spinning quite fast, so probably that was the roll rate sensor.

That would perfectly explain why the lift off looked normal for a few seconds and then the system went into positive feedback, trying to correct a slight roll but instead speeding it up.

0
0

I get what you are all saying about design, but...

...there has to come a point when you also need to hire competent people.

In a prefect world you could design something that is fool proof, but you should always remember, "if it is something any idiot can do, only an idiot will do it."

1
1
Boffin

up<--->dn

Infallible marking system.

11
0
Silver badge

Re: up<--->dn

It is only infallible if the use state of the 'thing' corresponds with the alignment markings on the part. Or, up during assembly may not be up when in flight.

1
0
Joke

Re: up<--->dn

Oh dear. I forgot my own infallible marking system...

<sarc>

Infallible marking system

</sarc>

<hint>

Try reading the title upside down.

</hint>

7
0
Silver badge

Re: up<--->dn

Clever, I like it :)

Took me a moment and your hint, but that was worth it

3
0

What about pre-flight tests?

Any sensible in-flight systems (not to mention remote telemetry setups) ought to have picked up on anomalous sensor readings and aborted before the motors were even ignited.

Sounds like failures all along the line, from sloppy design, through sloppy assembly.

0
0
FAIL

Re: What about pre-flight tests?

Err.. the sensor was apparently a vertical motion sensor. So until the thing actually starts to move, no-one knows there's a problem.

0
0
Flame

Re: Er, Fail Comment Fail

If the only way to test your equipment is to press the big red button ...

... I'll just wait to catch the next flight, thank you.

1
0
FAIL

Re: What about pre-flight tests?

...and if they've got a brain cell, there will be multiple redundancy. So if one sensor says "down" and all the others say "up", the on-board computer marks the anomalous sensor defective and ignores it unless or until it reliably returns to service or the OBC's telecommanded to bring it back into the fold.

Or, as in this case, not.

I write on-board software for space systems, so I do have a little insight into this.

0
0
Silver badge

Re: What about pre-flight tests?

Redundancy adds weight, and rockets are VERY weight-sensitive, meaning you run the risk of it being too heavy to fly.

0
0

Page:

This topic is closed for new posts.