back to article Computer error triggers mass rocket launch

Americans love their fireworks on Independence Day, but it is possible to have too much of a good thing. That's what spectators got on July 4 in San Diego, California, when an errant computer triggered every rocket in the city's annual display to launch at once. The pyrotechnics were meant to last 18 minutes. Instead, the …

COMMENTS

This topic is closed for new posts.
  1. Phil Endecott

    Last time...

    Last time I read something like this, the twits had entered a whole load of timing data thinking the format as hh:mm, when it was actually mm:ss. Hence a display 60X faster than expected. Seems about the same speedup as this incident.

    1. Anonymous Coward
      Anonymous Coward

      Re: Last time...

      Timing for these is usually hh:mm:ss:ff, driven by SMPTE LTC timecode from the audio source.

      However, even if everything is correct in the pyro sequencer, if somebody hits "fast forward" on the timecode generator while the system is armed...

      Anon because by the grace of God go I. Big public building openings? Oh yeah. Not nervous. At all. Everything's ready, it'll go fine.

    2. Anonymous Coward
      Anonymous Coward

      I like a good bang!

    3. JeffyPooh
      Pint

      Re: Last time...

      That doesn't make any sense. If the time format was assumed to be HH:mm then the plan would have a temporal resolution of one minute. That would make for a rather boring show. So it's a non-starter as a possible explanation.

      1. Anonymous Coward
        Anonymous Coward

        Re: Last time...

        I heard from a friend of a friend of a mate of the guy who worked at the Oban event.

        It was not an HH:mm against mm:ss confusion as that would have been extremely obvious. It was a mm:ss against ss:ff mixup.

        This is also one of the reasons why most timecoded pyro operators want at least 20 seconds of good timecode before the first 'fire' to give them some time for an "Ohshit" moment and pull the key!

        Anon to protect the red faces.

    4. James Micallef Silver badge
      Terminator

      Re: Last time...

      I like the way the company statement tries to take cover behind public ignorance of computers by saying "it wasn't human error, it was the program that malfunctioned", when really the malfunctioning program is because it was badly programmed by a human.

  2. James O'Brien
    WTF?

    ????

    Some sort of computer virus caused this? I smell bullshit from someone who needed something to blame besides their own incompetence. My guess is they probably had the display to set based off the computer clock and the clock was set wrong instead of using a good old fashioned timer. And even if they were using a timer script they probably dropped a couple 00's.

  3. Colin Miller

    Happend in Oban for Guy Fawkes night

    http://www.bbc.co.uk/news/uk-scotland-15611160

    1. smudge
      Happy

      Re: Happend in Oban for Guy Fawkes night

      Yup - that's the first thing I thought of.

      Where Oban leads, San Diego follows eight months later :)

  4. Anonymous Coward
    Anonymous Coward

    Virus? I'm gonna go out on a limb and guess a problem with the keyboard->chair interface.

    1. ElReg!comments!Pierre

      Yes, probably

      "must be a virus" is the modern age equivalent of "the dog ate it". get it for all kind of things, from data deletion to forgotten passwords.

      Last time it was for the inability to select the right printer queue. Apparently a "virus" (of which no trace could be found upon inspection) caused the change of the default printer queue hence much whining about the network being broken. I know it's bollocks, they know I know it's bollocks, but they keep trying. It has become a social convention, a bit like "how do you do" and "have a nice day". Meaningless converstation lubricant.

      "we have a virus" actually means "can you come and show me how to find my own arse with the help of a map, a compass and a lighthouse. Again."

      1. Euripides Pants

        Re: Yes, probably

        "can you come and show me how to find my own arse with the help of a map, a compass and a lighthouse. Again."

        Now, if you could ram the lighthouse up the clueless twit's arse....

    2. Old Handle
      Terminator

      You know what Agent Smith says.

      1. ElReg!comments!Pierre

        You know what Agent Smith says.

        Can you feel it Mr. Anderson?

  5. Lord Elpuss Silver badge
    FAIL

    Virus

    Yeah that'll be it. Nothing to do with programming errors or anything else that might have been caused by them, this clearly has virus written all over it.

    Idiot.

  6. edge_e
    Boffin

    Repeat after me

    The computer does what it's told to by the program

    If the computer does something unexpected it's usually because the programmer fucked up

    1. Michael H.F. Wilkinson Silver badge
      Boffin

      Re: Repeat after me

      The programmer OR the user fucked up.

      (not XOR, btw)

      My bet would be on the user. But, as I write programs, it would be, wouldn't it?

    2. LaeMing
      Happy

      Re: Repeat after me

      Computers do exactly what they are told. They do it faster than a human can realise they told it the wrong thing.

      1. bazza Silver badge

        Re: Repeat after me

        Forget computers, it's photocopiers that I don't get on with. It wasn't so bad ten years ago, pressing the wrong button would waste only a few hundred sides of paper. Now the damn things are so fast they spit out thousands and thousands of wasted sheets in the blink of an eye, all neatly collated and stapled.

  7. Aaron Em

    'Program glitch' my shiny pink arse!

    The eventual "glitch" will turn out to be whatever is the west-coast equivalent of a well-lubricated "Hey, y'all, watch this!", said revelation being swiftly followed by the quiet departure of one or more former pyro technicians.

    You heard it here first.

  8. Khaptain Silver badge
    Happy

    That was unique........lol

    Just to be on the safe side that could have added that it was probably Anonymous that plotted this and the virus was called the Gunpowder.Plot.

    Or maybe they forgot to turn on their FIREworksWALL

  9. Michael H.F. Wilkinson Silver badge
    Mushroom

    This is just a practice run for the RotM

    Next time they will do it using ICBMs!!!

    We're doomed, DOOMED!!!!

  10. Graham Marsden
    Facepalm

    I bet...

    ... the audience were thinking "Wow! If that's how it starts, what's the rest of it going to be like?! [Long pause] Err..."

  11. Anonymous Coward
    Anonymous Coward

    Someone's selected the wrong time-code when loading the scripts in to the firing system, which is what happened at Oban.

    1. Anonymous Coward
      Anonymous Coward

      You sir read the same forum as I do

      and it does seem likely doesn't it?

  12. Stevie

    Bah!

    Everyone is being a bit harsh - clearly the "spokesman" was on the spot and not terribly computer literate, using "virus" for "glitch" from the phrasing there.

    Coordinator at the optimum viewing point with a radio and human beings with manual ignition systems in the barges is the way to go, because if Mr Boss goes off-plan and says "everyone light everything at once" the bargees will say "No".

    Just because you can put a computer into your event doesn't mean you should.

    1. ElReg!comments!Pierre

      Re: Bah!

      "Coordinator at the optimum viewing point with a radio and human beings with manual ignition systems in the barges is the way to go, because if Mr Boss goes off-plan and says "everyone light everything at once" the bargees will say "No"."

      If I remember correctly they make damn sure that no soul remains aboard the barges at the time of firing. They even triple-check. Too dangerous.

      Plus, big modern fireworks are very precisely timed ballets, often needing the firing of a dozen devices situated on separate barges at the exact same time, for example. The time between firings is also extremely precisely controlled to ensure that a rocket blooms in the exact center of the aftermath of the previous one for example. Humans just can't react that fast/precisely, especially not over a walkie connection. The way it used to be done was by using very precisely calibrated fuses but that's more much prone to errors and mishaps than a computer and electrical ignition.

      Modern fireworks are actually an almost perfect use case for a computer.

      1. Jon Green
        Mushroom

        Re: Bah!

        They certainly made sure that no soul remained aboard _those_ barges!

      2. Stevie

        Re: Bah!

        "Plus, big modern fireworks are very precisely timed ballets, often needing the firing of a dozen devices situated on separate barges at the exact same time, for example. The time between firings is also extremely precisely controlled to ensure that a rocket blooms in the exact center of the aftermath of the previous one for example."

        Makes you wonder how they ever managed to pull the trick off for the three hundred years or so before computers were ready to take up the slack.

        Or not, as in the case of the San Diego Fireball of Unentertainment.

        I imagine you'll be telling me next why the pyramids require the intervention of UFOs to get built.

        1. ElReg!comments!Pierre

          Re: Bah!

          > Makes you wonder how they ever managed to pull the trick off for the three hundred years or so before computers were ready to take up the slack.

          As I stated: very precisely calibrated fuses. Not as precise or safe as electronic system but it did the trick

  13. Naughtyhorse
    Trollface

    it's iran im telling you

    It's stuxnet.

    Hitting the infidel where it hurts.

    Allah Akbar

    told you something like this would happen

    1. bazza Silver badge

      Re: it's iran im telling you

      Iran, on Independence day? Surely not. Most likely to be the Brits don't you think?

      1. auburnman
        Angel

        Not us, guv'nor

        Us? We like a good fireworks display as much as anyone else. We also like to laugh at someone else'e ballsup a little more than we should probably admit, but it's not as much fun if you deliberately cause them to eff up.

        On the subject of the "must be a virus" guy, I have a fair bit of sympathy for him as he suddenly became the centrepiece for a highly public SNAFU on a major holiday event. If it happened to me I'd be probably dropping all the FUD and chaff in my verbal arsenal to buy time for a retreat and regroup.

      2. edge_e
        Coat

        Re: it's iran im telling you

        All your rockets belong to us dear chap

  14. PyroMoth
    Mushroom

    RE: Bah!

    I will assume you have never stood in close proximity to a large scale show, or had to fire a show manually to rapid cues.

    Just like the world of lighting, computerised firing controllers are ubiquitous. They are far safer and can give much more accurate choreography. Thousands of shows are run in the same way each year without issue.

    I'm pretty sure someone just made a mistake under pressure on the busiest day of the year, and the script got messed up as a result. It will have happened to most companies one way or another. No point lying.

    1. Stevie

      Re: RE: Bah!

      "I will assume you have never stood in close proximity to a large scale show, or had to fire a show manually to rapid cues."

      Then you will assume incorrectly.

      Makes you wonder what your other assumptions about this San Diego Fiasco are worth, eh?

      1. This post has been deleted by its author

      2. This post has been deleted by its author

  15. Phil Koenig
    FAIL

    Advanced WiFi control system

    How much you want to bet the person who designed the control system used a $39 WiFi hotspot with encryption turned-off? Maybe it was the same intellectual that designed the retail point-of-sale system for TJX in the States...

    http://www.schneier.com/blog/archives/2007/10/tjx_hack_blamed.html

    1. Richard 12 Silver badge
      FAIL

      Re: Advanced WiFi control system

      Nobody even considers using WiFi for this.

      This kind of event needs around 10ms accuracy, and WiFi cannot possibly do that as its latency varies wildly.

      WiFi is great for email and surfing the web, it's ok for buffered video but it's utterly useless for anything that needs even mildly accurate timing.

      1. ElReg!comments!Pierre

        Re: Advanced WiFi control system

        > Nobody even considers using WiFi for this.

        What you meant was "Nobody _with actual knowledgẹ_ even considers using WiFi for that."

        Stevie up there seriously suggested using fleshlings on the barges with manual ignition systems, walkie-takies and a central coordinator. The latency of that being in the seconds range obviously (for road safety purposes the reaction time of a human is estimated to be roughly one second; add the "stellar" clarity of radio communication to that...)

        1. Richard 12 Silver badge
          Happy

          Re: Advanced WiFi control system

          Actually, while humans have a long latency to unexpected events, given suitable warning and a 'click track' most of us mere fleshlings can reliably hit a cue within 50ms, and some people down to 10ms.

          It's all down to the rhythm of the dance.

  16. Kev99 Silver badge

    My data, your computers? Uh-uh!

    And the pundits expect me to put my proprietary & other data in the hands of someone else's computers "in the cloud"? I don't think so, Tim.

  17. stucs201

    Is it really that much of a screwup?

    From the background cheering the audience seemed to like it.

    (whether that lasted when they realised how short it was is another matter)

  18. Peter Fox
    Thumb Down

    What could possibly go wrong?

    As clocks go 'tick' and cows go 'moo' so programmers go 'WCPGW". Did no programmer not think 'maximum rate of firing instructions is X'?

  19. Esskay
    Pint

    Not entirely a bad thing

    As any fule kno, the quicker the fireworks are over, the quicker you can get back to drinking.

    1. I ain't Spartacus Gold badge

      Re: Not entirely a bad thing

      Excuse me, but why are the two mutually exclusive?

      If the bar's not serving, then that's what the Good Lord invented the hip flask for. Admittedly the fireworks displays I've been to in England tend to be New Year and November. So a warming nip of whisky doesn't go amiss.

  20. qt101
    Joke

    USA's 'Premature Ejeculaunches'? On July 4th ;-D

    They've always gotta get to the big Bang fast.

    Re: Virus in our software. Pfft please! lol

    Next Excuse will be: We've had specialist analysist's concluded that we've indeed been Pwn'd by a Hacker!

  21. amanfromMars 1 Silver badge

    Cogitant ergo sunt .. and an APT rallying call in these changed days of strange ways and memes?

    Neil, Hi,

    A word of advice, which you can if you want to, classify as insider information and a hot tip ..... don't put any money you cannot afford to lose, nor any shirt which you want to keep, on a bet that would support the article's sub-headline ....... Rise of the machines thwarted ... for now

    Any latin scholars [are you an El Reg reader, Boris J?] out there who can supply the correct translation into Latin of ..... "They think therefore they are" , which of course is a play on that well known classic line ..... Cogito ergo sum.

  22. Paul Hovnanian Silver badge
    Mushroom

    Its only fireworks

    It could have been worse.

    "How about a nice game of chess, Doctor Falken?"

    1. Anonymous Coward 15

      Re: Its only fireworks

      Was waiting for that one, and if I didn't see it I would have posted it myself.

  23. Anonymous Coward
    FAIL

    Ahhhhhhh ............................... well.

    The show is over, C'mon kids - lets go home.

  24. beep54
    Devil

    Cool!

    As a 58 year old American I have to say that THIS was the 4th of July spectacle that I have been waiting my whole life to see :)

  25. RocketBook
    Joke

    I told you

    That Y2K bug would cause problems - one day. Surprised they never thought of that one.

  26. Test Man
    Stop

    Didn't this happen in the UK a few months ago also?

  27. Apple][Guy
    Mushroom

    PE

    Well there's a case of PE for you...

    1. Anonymous Coward 15
      Joke

      Re: PE

      "Why aren't you wearing any trousers?"

      "Because I just came in my (under)pants."

  28. Anonymous Coward
    Anonymous Coward

    A virus did it

    It's probably been remarked upon enough above, but I'm going to use that excuse the next time I screw up. I'm just kicking myself for it having taken me this long to realise how to explain introducing a fatal bug into the system!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: A virus did it

      See what happened when they tested "turning off the DNS Changer servers"?

      Damn feds ruined the celebrations

  29. Anonymous Coward
    Anonymous Coward

    "nor was there any human error"

    Fucking idiots of course it was human error the computer did exactly what it was told. the fact that the stupid computer operator typed the time values in tenths of seconds in to a program taking instructions in miliseconds was definitely "human error".

  30. Crisp
    Terminator

    It was obviously Skynet.

    Skynet obviously took control of the rocket launch system and tried to wipe out the human race.

    Unfortunately, due to an administration cock up, Skynet ended up with the launch codes for San Diegos firework arsenal instead

  31. Tom_

    Result

    Fireworks can be nice to watch, but not for quarter of an hour.

    1. Dan 10

      Re: Result

      Stiff neck

      I've always wondered what a display would look like if they set everything off at once.

    2. asdf
      Mushroom

      Re: Result

      Quarter of an hour is short by our standards as many go closer to a half hour. SD was being cheap.

  32. Sceptic Tank Silver badge
    Windows

    Visible from the moon?

    The party was great! We had a blast!

    But, yes, it's sad when reading a report about a fireworks display takes longer than the display itself....

    Maybe the pyrotechnics guy upgraded his old Microsloth Windoze operating system to something a bit zippier just prior to the show and the program ran in a fraction of the time it did before. That's probably why the chap though he had a virus and observed program glitches before too.

    Or something...

    1. hplasm
      Coat

      Re: Visible from the moon?

      So *thats* what happens when you press the 'TURBO' button on the PC...

  33. unitron
    Mushroom

    "There was a malfunction of the firing systems which ignite the fireworks,"

    You mean there were some that didn't go off?

    Looks to me like the firing sysems ignited them just fine.

  34. Anonymous Coward
    Anonymous Coward

    Very much like my love life this. All over in 15 seconds ...

  35. kain preacher

    next excuse. Well some idiot fired off a gun into the fireworks and it went boom.

    I'm sorry but say it's a virus is the equivalent of calling sick and your boss sees you on the telly at a foot ball game.

  36. LordHighFixer
    Facepalm

    Most likely

    The display had a test phase that is supposed to run before the show starts. You know, something that tests the firing control circuits right before the show that is supposed to run, with the "fire" switch in the off position. Then after the initial test and announcement the switch is switched to the "on" position for the show.

    I can imagine the control operator starting the program, like he or she has done a dozen times before, looking over and seeing the switch in the "fire" position. Trying to jump for it and having an "oh crap" moment as he hears the first detonation.

    Honest, it was a hardware glitch....kinda...

  37. Comments are attributed to your handle
    Black Helicopters

    Has anyone else read "Zero Day" by Mark Russinovich?

    It's starting.

  38. Alan Esworthy
    FAIL

    "Glitch". Ugh.

    "Although the company has yet to determine the exact nature of the glitch that caused..."

    It's bad enough when non-techies use the term "glitch" but when a Reg writer use the term I cringe. A "glitch" has the same sense as "just one of those things" or "these things happen" implying that no one was responsible, if not outright inevitability.

    This wasn't a "glitch". It was an "error". Somebody screwed up: designer, coder, packager, installer, tester, wire crew, user, SOMEbody.

  39. Zedsquared
    Coat

    Good to see the Reg upholding the tradition

    of the media calling all big display fireworks "Rockets" ... They're "Shells" goddammit! Nobody uses rockets any more on a big public display, too much worry about where the sticks end up.

    ... Mine's the anorak with the bits of igniter wire in the pocket and burn holes patched with gaffer tape.

    1. Aaron Em

      Re: Good to see the Reg upholding the tradition

      No sticks -- those are mortars? Wow, y'all are a lot more serious than I ever imagined...

  40. The Grump
    Pint

    The Boss had to do a last minute inspection...

    although the BFOFH (Bastard Fireworks Operator From Hell) told him everything was programmed correctly. When the Boss was checking the timing interface, he said to himself "This can't be right", and "corrected" the timing.

    Now the Boss has to determine who's fault it was (obviously not his own fault). It won't be pretty at Mission Control, especially when the CEO gets the company stock reports. I would love to see how the BFOFH would cover his arse by implicating a beancounter from accounting, who was looking into "unnecessary expenses" at Mission Control.

    Beer, because they'll need a lot of it after this disaster.

  41. Anonymous Coward
    Coat

    Reminds me

    of heterosexual sex before I became a lesbian (sorry, here come the down votes)

    1. asdf

      Re: Reminds me

      Nah lesbians are hawt. Still as usual its the internet where men are men, women are men and children are the FBI (MI5?).

    2. ElReg!comments!Pierre
      WTF?

      Re: Reminds me

      > Reminds me of heterosexual sex before I became a lesbian

      I feel sorry for your poor experiences but I seriously hope tha was not your only motivation for switching teams.

      As a heterosexual male I have experienced _plenty_ of really crappy and insatisfactory (for me) heterosexual intercourse. Bad sex usually has nothing to do with the relative genders of those involved. It's usually simply down to either lack of experience ("what am I supposed to do with that?"), selfishness ("oh yeah baby that's good! Good night now. *SNORE*") or lack of involvment ("Sorry hun I REALLY have to answer that."). There are other reasons but if you look closely they often boil down to one of these 3.

      Not that there is anything wrong with being a lesbian. As a heterosexual male I've always understood lesbians better than heterosexual women. What's not to like in a female body?

      But all this would probably be best discussed in another place. This article was about dry things that go boom.

      > (sorry, here come the down votes)

      Why the fuck would you think that? I was tempted to downvote you just for that remark, then I remembered that some people care about votes for some unfathomable reasons so downvoting might be considered rude.

      1. Aaron Em

        "What's not to like in a female body?"

        Periods -- god almighty, of all the things that make me glad I'm male.

        As for the rest, I suppose it must be true what I've heard about Frenchmen.

  42. Stephen W Harris
    Joke

    I blame the leap-second...

  43. Identity
    Mushroom

    Trivia

    I've heard that the nickname —even before the event — was "The Big Bang Boom."

  44. William Higinbotham
    Linux

    Reverse Polarity without Diodes

    Maybe the leads where reversed. Without proper diode protection, if I read it right, the first signal will set them all off possibly - ground loop effect.

  45. William Higinbotham
    Devil

    I think I saw the Gold Particle in the show

    All these big machines and we could do it with pyrotechnics <Grin>

    Now to sift through the data from the videos and submit to Computing in High Energy and Nuclear Physics June 22, 2012 • New York • United States

This topic is closed for new posts.