back to article Atlantis computer goes down: Fixed by 'nauts

Commander Chris Ferguson and pilot Doug Hurley have fixed space shuttle Atlantis' General Purpose Computer (GPC) 4, which clapped out last night requiring transfer of its duties to another of the shuttle's quintet of GPCs. The pair have reloaded software into the unit and it has "been added to the common set of GPCs and is …

COMMENTS

This topic is closed for new posts.
  1. Ben Holmes
    Happy

    A candidate...

    ...for a future edition of El Reg's 'This Old Box', perhaps?

  2. Alister Silver badge
    Trollface

    Turned it off and on again?

    See, you lot may mock, but it works a lot of the time...

    Oh, and after "Windows for Warships" (http://www.theregister.co.uk/2009/01/05/windows_for_warships_hits_type_23s/) I was hoping to see Windows for Spaceships, I really was...

    1. Mike Flugennock
      Thumb Up

      re: This Old Box?

      Just upvoted your post, with enthusiasm. "That's affirm", as they say in Mission Control.

      Man, that'd be the mother of all "This Old Box" columns.

      While you're at it, Vultures: if you're going to do a "This Old Box" column on the Space Shuttle GPC, then you absolutely _have_ to do one on the Apollo computers: the Guidance & Navigation computer aboard the CM, as well as the LM onboard computer. There's some interesting stories about how the fussy LM computer aboard Apollo 14 nearly prevented them landing until they figured out how to hack it -- yeah, that's right, hack it -- in a procedure involving geeks on the ground hacking a backup LM computer in Houston and radioing up the hacks to Shepard and Mitchell in lunar orbit. Talk about "BOFH And Proud".

      1. Andus McCoatover
        Windows

        MORE!!! @ Mike Flugennock

        Hacking Apollo 14 computer? Wow.

        We want more info (Links would be great...). That's the kinda stuff we love!

        1. Mike Flugennock
          Boffin

          a few Apollo 14 LM computer hack links, for starters...

          http://www.universetoday.com/42045/apollo-14/ :

          "After separating from the command module in lunar orbit, the LM Antares also had two serious problems. First, the LM computer began getting an ABORT signal from a faulty switch. NASA believed that the computer might be getting erroneous readings like this if a tiny ball of soldering material had shaken loose and was floating between the switch and the contact, closing the circuit. The immediate solution was to tap on the panel next to the switch. This did work briefly, but the circuit soon closed again. If the problem occurred after the descent engine fired, the computer would think the signal was real and would initiate an auto-abort, causing the Ascent Stage to separate from the Descent Stage and climb back into orbit. NASA scrambled to find a solution, and determined the fix would involve reprogramming the flight software to ignore the false signal..."

          http://en.wikipedia.org/wiki/Apollo_14 :

          "After separating from the command module in lunar orbit, the LM Antares also had two serious problems. First, the LM computer began getting an ABORT signal from a faulty switch. NASA believed that the computer might be getting erroneous readings like this if a tiny ball of solder had shaken loose and was floating between the switch and the contact, closing the circuit. The immediate solution—tapping on the panel next to the switch—did work briefly, but the circuit soon closed again. If the problem recurred after the descent engine fired, the computer would think the signal was real and would initiate an auto-abort, causing the Ascent Stage to separate from the Descent Stage and climb back into orbit. NASA and the software teams at MIT scrambled to find a solution, and determined the fix would involve reprogramming the flight software to ignore the false signal. The software modifications were transmitted to the crew via voice communication, and Mitchell manually entered the changes (amounting to over 80 keystrokes on the LM computer pad) just in time."

          http://history.nasa.gov/afj/compessay.htm :

          From the Apollo Flight Journal, an overview of Apollo on-board computer systems. No wussy-assed GUI OS's for the old Right Stuff boys.

          1. Andus McCoatover
            Windows

            Thanks, Mike. Greatly appreciated.

            As you say, the "Right Stuff boys".

            Fascinating. Real engineering again.

            (of course if it were Microsoft software, it could only be fixed on a Tuesday)

  3. TeeCee Gold badge
    WTF?

    Hmm...

    So they've rebooted it and it's now fine, but what actually caused it to go titsup is a mystery?

    Sounds like their 400,000 lines of code have got something in common with the rather larger quantity referred to after all.....

  4. Pete 43
    Pint

    Three finger salute?

    Turn it off, turn it on. Bingo!

  5. Anonymous Coward
    Happy

    True embedded

    For those that read the DailyWTF this is true embedded computing.

  6. Jeremy 2
    Coat

    Hello IT...

    So they turned it off and on again, then?

    1. John Smith 19 Gold badge
      Happy

      @Peter 43

      "Turn it off, turn it on. Bingo!"

      I think the usual NASA term is "Cycling the breakers"

      1. Mike Flugennock
        Boffin

        re: "cycling the breakers"

        http://www.universetoday.com/42045/apollo-14/ ...again, from their overview of Apollo 14 computer and landing radar issues. As usual, an emergency software patch introduces new bugs. Sound familiar...?

        "Another problem occurred during the powered descent. The LM radar altimeter failed to lock automatically onto the moon’s surface, depriving the navigation computer of vital information on the vehicle altitude and groundspeed. This was an unintended consequence of the software patch. After the astronauts cycled the landing radar breaker, the unit successfully acquired a signal near 15,000 m. Just in the nick of time. Shepard then manually landed the LM closer to its intended target than any of the other six moon landing missions."

        By comparison, it sounds like the Atlantis crew had it pretty easy.

  7. Lee Dowling Silver badge

    Tried, Trusted, Tested.

    Why would anyone even think that slapping a random chip up there would be a good idea? Yeah, for everyday entertainment, or casual use, or non-essential admin work, but for life-critical functions?

    Hell, they've *already* got four of the things doing nothing but double-checking each other's calculations, and it's an incredibly hostile environment (think what happens if there's a tiny bit of condensation - you can't just open a window, not to mention flying through varying EM fields, cosmic rays, and potentially unstable power - and tech support is several months away). By comparison, the liftoff was nothing.

    I'd much rather they were using these old clunkers, presumably with chips that have 30+ year errata's, billions of hours of real-world testing in commercial and industrial environments and production lines established for decades, than trying to slap an Intel with an undiscovered FDIV bug up there. Plus the old chips were predictable to an extreme degree (down to the individual cycle). Modern chips are multi-core, have vast caches, unpredictable bus intersections and all sorts of problems. That's not what you want. And 1.4MHz is more than fast enough for doing anything critical, it's going to be vastly outweighed by the time for that action to actually have a physical effect.

    1. James Hughes 1

      That as maybe...

      But will it play Crysis?

      1. Anonymous Coward
        Facepalm

        Re: That as maybe...

        "But will it play Crysis?"

        They're in space! Why would they want to play First Person Shooter #378? SPACE!

        (Although I'm sure coming generations, should they get the chance to orbit the planet, will be more interested in checking their "Facebook 3" or "Google++" feeds than actually taking a moment to appreciate the view or the achievement or anything like that.)

      2. Sharpy86
        Coat

        think with portals

        Just needs to play portal 2.... I'm in spaaaaaa-

        Yeah, i'll get my coat

    2. spodula

      Heck,

      They are still making RCA 1802's for certain military applications, and they were originally designed in 1976. Of course, they were the first real microprocessors that were radiation and ESD hardened.

      Of course its not fast, but its much likely to continue working in conditions that would cause most other processors to just die instantly.

    3. John Smith 19 Gold badge
      Happy

      @Lee Dowling

      "and production lines established for decades,"

      Not really. But the instruction set is more or less a stock S/360 mainframe and AFAIK they use *standard* mil spec TTTL chips so you can repair it if you have such parts (eBay is NASA's friend as well).

      The *real* cost is in all the certification tests to prove that it works in this environment. In terms of gate count you could probably put the whole thing on 1 ASIC (*including* the RAM even at mil spec gate densities).

      Embedded designers usually work on the principle of fast enough to get the job done.

      BTW These are the *upgraded* GPC's from the mid 90's with DRAM memory (replacing core store) and the whole thing ( CPU + *monster* I/O subsystem) in one box instead of two.

  8. Captain Scarlet Silver badge
    Paris Hilton

    Erm

    Isn't the shuttle design quite old anyway?

    Paris because I will probably be made to look really silly in a moment or to.

  9. The Serpent

    If it ain't broke, don't fix it..

    If it ain't broke, don't fix it it essentially the safest thinking here - fine I understand the compulsion in that.

    However, that processing speed and memory requirement are easily achievable in a single chip which is surely going to be the most stable possible design, both for mechanical vibration and as far as anti-radiation casings without a huge weight penalty go.

    In fact, the main radiation defense in the NASA article says it is a "memory scrubber", in other words it is implemented in software and/or the memory controller meaning they deliberately allow exposure to radiation and then bypass the effects.

    It should be possible to recreate the system in it's 'built for purpose' function but utilising some modern advantages in form. I'll send them an email so they can get on with it in time for the next one.

    Oh. Hang on...

  10. IglooDude

    Hmmm

    "no one straps the latest-and-greatest desktop computer..."

    Surely there's a little daylight between "the latest-and-greatest desktop computer" and a computer with the processing power and memory of a circa-1980s IBM 286? Not that it matters now I suppose *sob* and it is simply representative of the shuttle's late-70s construction. But it is promising in a way (for the likes of Elon Musk), that one could replace it with one solid-state smartphone containing fifty thousand times more processing power (and an ability to play mp3s so as not to require Elton John to pipe his music up to the astronauts live) and so much smaller and lighter that one could insulate it enough to enable it to survive conditions that would turn the astronauts into reddish mush.

    1. GettinSadda
      Angel

      Gee

      Gee, I wonder why this mid-eighties spacecraft contains a mid-eighties spec computer?

    2. Anonymous Coward
      Facepalm

      Re: Hmmm

      "But it is promising in a way (for the likes of Elon Musk), that one could replace it with one solid-state smartphone containing fifty thousand times more processing power (and an ability to play mp3s so as not to require Elton John to pipe his music up to the astronauts live) and so much smaller and lighter that one could insulate it enough to enable it to survive conditions that would turn the astronauts into reddish mush."

      Do you really think they use this gear to play music with? Besides, astronauts have taken more modern kit into space routinely, and if you look into the early Shuttle missions you'll see kit that blew away contemporary microcomputers more widely used on Earth. For example:

      http://en.wikipedia.org/wiki/GRiD_Compass

      The gear in question, however, absolutely has to function with complete reliability. It can't crap out with a blue screen half way to orbit: such situations are hazardous enough already.

    3. Ru
      FAIL

      Re: and an ability to play mp3s

      Why do critical safety and avionics systems need to play MP3s? Why do they need 50k times more processing power? The system has proven quite adequate for surface to orbit (and back) travel many, many times. Upgrading would be frivolous, pointless and potentially hazardous; there's nothing to gain and everything to lose.

      Astronauts can and do use significantly more modern and powerful computers. I doubt they're using flight control systems to send tweets.

      1. IglooDude

        Ummm

        The mp3 bit was a joke, just for the record.

        1. Andus McCoatover
          Coffee/keyboard

          The mp3 bit was a joke, just for the record.

          "Just for the record???"

          New keyboard, please.

    4. Anonymous Coward
      Anonymous Coward

      Why?

      Why spend millions environmentally testing and installing something when what they have does the job just fine.

      Also: Denser components and more power = more chance of getting zapped by a cosmic ray. It's an occasional problem here on Earth, and I imagine that it's a more likely event up there.

      If it ain't broke, don't fix it.

      1. The Serpent

        Why??

        Spreading things out increases their chances of being hit by something, not putting them closer together. Also, what has power got to do with it?

        1. Anonymous Coward
          Anonymous Coward

          "Spreading things out increases their chances of being hit"

          Yes and no

          Chances of being hit increase with larger size but chances of the hit causing malfunction increase with smaller components ( i.e higher densities) because each element will require fewer electrons to change state.

          1. The Serpent

            "Spreading things out increases their chances of being hit"

            You're comparing apples with oranges there - talking about size and then comparing it with the criticality of being hit.

            Everything on the shuttle's GPC's is critical as it is designed for purpose. NASA know that and designed in a system to bypass the inevitable damage. My thought about re-packaging the GPC's into a smaller form improves the situation by being harder to hit. As a re-package, it should include the "memory scrubber" already present to deal with those unlucky whacks. The risk is reduced and the countermeasure is equally as effective. Then you can go on to all the advantages (increased supplies/cargo payload, etc), perhaps even ease of replacement, which reminds me..

            I note you have switched to a material argument, rather than continuing with power or size. If you have any specific data about how easily the state of any given material is changed by contact with forms of radiation commonly found during shuttle orbits, please do point them out as it's the only thing I'd like tidied up.

    5. Peter Mc Aulay

      Re: Hmmm

      They do have modern laptops up there, but they don't let them fly the ship, obviously.

      Your car's engine electronics don't play mp3s, either.

  11. Mark McNeill
    Coat

    Hope they tried the Help facility first.

    It looks like you're transferring cargo from the space shuttle to the ISS.

    Would you like help with that?

  12. Johnny G
    Alert

    "Open the pod bay doors GPC4"

    "I'm sorry, Dave. I'm afraid I can't do that"

    1. Benchops
      Flame

      What no astronaut wants to hear:

      "I've just picked up a fault in the AE35 unit. It's going to go 100% failure in 72 hours."

      1. Anonymous Coward
        Coffee/keyboard

        Re: What no astronaut wants to hear:

        "Would you like to send a problem report to the Windows Service Center?"

  13. John Savard Silver badge

    It's A 360!

    I had to look elsewhere, though, to find out that those computers are IBM AP-101 computers, part of the System 4/Pi line from IBM and compatible with the System/360 architecture.

    1. This post has been deleted by its author

    2. Anonymous Coward
      Anonymous Coward

      360 memories

      An IBM 360? Where's my green card?

      (And, while we're at it, we need an "old codger" icon.)

      Anonymous, because I'm showing my age.

  14. Andus McCoatover
    Windows

    Sorry, folks...

    I think it's absoutely bloody fantastic.

    Lesson in reliability and code efficiency. Computer absolutely fit-for-purpose. Doesn't play games, surf the web, let you watch porn. It flies the shuttle, exactly what it was designed for.

    Remember Apollo 11? How much "intelligence" that machine had? Nowadays we slide a gig. of RAM into our machines, and think nothing. They had "core rope".

    Times change, but acheivements do not.

    1. Charles 9 Silver badge

      Not knocking it.

      I'm pretty sure this was as robust a piece of 80s tech as they could get, but considering that computing has progressed by a number of orders of magnitude since then, I'm pretty sure one could develop a similar design today in a much smaller package: complete with the necessary redundancies to account for EMI.

    2. John Smith 19 Gold badge
      Happy

      @Andus McCoatover

      One of the developers kindly sent me his personal commentary of one of the few books written on it.

      36KB of "Woven rope" ROM with 1KB of RAM implemented by ultrasonic delay lines triplicated and majority voted for reliability in a package weighing roughly 90lb. I also thought the electrochromic displays on the instruction panel were pretty cool, but they never caught on.

      Increasing the instruction set size by deliberately causing a math overflow on a variable then *executing* it was my second actual encounter of the idea of self modifying code (the first was the Bell Labs Blit terminal). I'm not sure what other sacred cows of modern software development MIT slaughtered to get the job done.

      It's processing speed was *literally* (at 32KIPS ) the same as a modern pocket calculator.

  15. Johnny Canuck

    This is not a title

    Fdisk, format, reinstall.

  16. Anonymous Coward
    Boffin

    but wait....

    I remember they started packing a laptop on missions a few years ago. The laptop could run the whole shuttle if all the computers failed.

    There is no point to this comment. Just another useless tidbit of info.

    1. Anonymous Coward
      Anonymous Coward

      Apollo

      I think they packed a HP programmable calculator in case the on-board computer failed

  17. Wile E. Veteran
    Coat

    360's Rock - especially 360/67's

    Slow by today's standards if you're talking about pure MIPS, but with a DECENT timeshare OS like MTS (Michigan Terminal System) which was the basis of computing at my alma mater (no, it wasn't Univ. of Michigan, it was Wayne State U. a few miles from Ann Arbor) it could support hundreds, if not thousands of simultaneous users all beavering away in FORTRAN G or H, Assembler/360 or even (shudder) COBOL, APL and a few other weird languages. I did physics simulations on one of these beasts. developing in FORTRAN G because it was cheaper to run then switching to FORTRAN H for the actual simulation runs, and no doubt the simulation would run much faster on my laptop. But my laptop can only support one user at a time (even under Linux due to lack of hardware support for multiple terminals), not hundreds.

    Reliability and uptime in "5-nines" territory and that was just the "ordinary" commercial model.

    1. Anonymous Coward
      Anonymous Coward

      "even under Linux"

      I think it should using SSH over the network or connections via the serial port or more likely USB/serial converter.

  18. Rob Daglish
    Joke

    Hardly a great advert though...

    Seriously, would you buy a car that the manufacturer described as "handling like an old truck on a washboard"?

    1. Andus McCoatover
      Windows

      ..."handling like an old truck on a washboard"

      Seriously, would you buy a car that the manufacturer described as ....

      Only on it's last trip....

  19. Wanda Lust
    Joke

    FUC up in space.

    From the NASA.gov article: "That's why the astronauts started carrying fold-up computers originally made by GRiD into space."

    Fold-up computer never made into the daily lexicon.

  20. Anonymous Coward
    Meh

    "Times change, but acheivements do not."

    I think that pretty much sums up the good and bad in the article.

    Good: 40 years ago a dream team of the worlds best engineers did something really amazing.

    Bad: 40 years later they are all retired or dead, and the nation is now so lame that it may never do anything that great again.

    1. Anonymous Coward
      Facepalm

      Yes, but..

      They could always sue for patent infringement if anyone else tries!

  21. John Smith 19 Gold badge
    Boffin

    Some GPC notes.

    It's good enough to get the job done.

    Which for hard core embedded system types (it fails, people die) is pretty much the key issue.

    The 5 GPCs also communicate with 25 Multiplexer De Multiplexers (MDM). These each have 16 slots for I/O boards but it's not clear if they can pre filter the data so the GPC's only have to deal with "interesting" data. I've never found a reference which says how many of them are fully loaded either (these boxes are pretty heavy). Honeywell (who make them) do a range of ISS hardware with boxes holding 4,8,12 and 16 but I'll bet they don't fit.

    The Shuttle was one of the first to use the 1553b LAN standard. The 25 links clock at just about 1mbs each. The later 1773b uses FO cable, could operate as a drop in or use a high speed mode of 20mbs each (it should be more lightning resistant which has been one of the reasons for launch scrubs).

    The whole thing could be done in a single ASIC even given that MilSpec hardware is typically 1-2 generations behind commercial density *but* they live in fear of *unreported* changes in the mfg process (Gate array cell design or fab sequence changes in small but *critical* ways, rendering the new stuff more sensitive to latchup/single event upset etc) taking out the whole unit at a critical moment.

    GPS's are programmed in a high level language called HAL/S, a language used nowhere else (a bit like the Shuttles aircon fluid but rather less toxic), which is a pity as it allows entry of sup-scripted and super-scripted variables on a standard terminal. More usefully it also had matrix and vector functions (handy if you do a lot of navigation, or computer graphics) and data types, along with real time scheduling functions. Quite nice for the early 1970s.

    The software used to loaded in phases (depending on flight stage) from a reel to reel tape drive but this went solid state about 10-20 years ago.

    Shuttle software development (done by what was IBM Federal Systems in Houston) was *the* model for the CMU Software Engineering Institute's CMM standard for how good software teams are. They also built the software for Skylab. They did not seem to use CASE and their key technique seemed to be the "Structured walkthrough." It is estimated they cost 10x the average cost of writing a line of code in the US. A lesson for all programmers perhaps?

  22. Sargs

    Sometimes upgrading something that works well enough can be beneficial

    I think I recall reading in Siddiqi's "Sputnik and the Soviet Space Challenge" that in the seventies, the Soviets managed to save something like over a tonne of mass on their Soyuz launchers by scrapping all the old analog electronics and control systems and replacing them with digital, so there's two sides to holding on to proven tech.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019