back to article IBM Australia didn't stress-test #censusfail router and blocked password resets

If Vulture South wasn't running short-staffed yesterday, we'd have stayed with the Senate Committee hearings inquiring into Australia's Census outage on August 9, and caught this gem: “If we had our time again we would have tested a hard power it off, power it on that router, that would have discovered earlier that we had that …

  1. ecofeco Silver badge

    Turn it off and back on

    You know, I've seen this actually work so many times I've stopped making fun if it.

    Oh, and "told you so".

    1. John Miles

      Re: I've seen this actually work so many times

      The main reason it works is because it resets everything into a known state and any issues related memory/resource leaks, conflicts, stale configurations or in memory data corruption are cleared away (leaks will generally require periodic reboots - I have resorted to scheduling reboots/restarts when clearing memory leak isn't easily solved).

      Often it would be possible to avoid the reboot (assuming you have sufficient permissions) - but it is much easier and quicker to tell most users to just restart the computer and doing so it will have cured most user immediate problems without getting "technical"

      1. Anonymous Coward
        Anonymous Coward

        Re: I've seen this actually work so many times

        Back in the day...

        Had a data-logger running on a Win95 PC (yes, that long a go) - but because Win95 had memory leaks and because they client never bothered to switch off the PC I had to set it to auto-reboot at midnight to reset everything.

        So not quite "switch it off/on to reset", but close enough...

    2. Paul Crawford Silver badge

      Re: Turn it off and back on

      But sometimes it does the opposite - you find that config was updated in memory and not saved, so it comes up broken. Either way, it is really stupid that they did not test a complete reboot/power cycle of the system.

  2. stratcat

    copy running-config startup-config.

    There. I fixed it.

    1. DonL

      I guess they figured it would sound dumb if they admitted someone simply forgot to save the configuration. And now people could draw the conclusion that it might be a router failure.

    2. Alan Brown Silver badge

      "copy running-config startup-config."

      Unfortunately in some Cisco routers, this doesn't necessarily mean that it will start.

      Ask me how I know this. :/

  3. Denarius
    FAIL

    Privacy and security

    so password reset was done overseas ? Whisky Tango Foxtrot cubed. So all those "privacy statements " I have endured while trying to deal with assorted Oz gov sites this week are as good as the paper they are not written on ?

    1. Halfmad

      Re: Privacy and security

      If they used IBM, probably.

      Honestly "Big Blue" wonders why people increasingly avoid it, simple - when they are trusted they royally balls it up.

      This census was a big deal, they cut corners to save a few quid. Aussie government should ban IBM from procurement for a 10 year period.

  4. Mark Simon

    Translation please

    “If we had our time again we would have tested a hard power it off, power it on that router, that would have discovered earlier that we had that reboot and configuration loading problem”

    I know roughly what this means, but is this what passes for a coherent sentence these days?

    1. david 12 Silver badge

      Re: Translation please

      That's what you get when spoken English is copied out as written English.

      Conversely, written English sounds pompous and obscure when spoken.

  5. gregthecanuck
    Trollface

    Fail Whale

    "IBM and the ABS managed to put in place another howler of a misconfiguration: “Island Australia” (which only partially worked) was designed to block traffic from outside Australia, but IBM was hosting the Census password reset facility offshore."

    OMG - I laughed so hard my wife was like "WTF?".

    1. Robert Carnegie Silver badge

      Re: Fail Whale

      ICBW, I thought that "Island Australia" was only created as an on-the-fly attempt to stop the DDOS in progress? So, mistakes were made, but this mistake happened after the fire started?

  6. Anonymous Coward
    Anonymous Coward

    Axiom vs Meme

    Nobody gets fired for buying IBM.

    vs

    Turn it off and on again.

    Fight!

    1. allthecoolshortnamesweretaken

      Re: Axiom vs Meme

      Looking at IBM's general performance, I think it is "Nobody got fired for replacing IBM with something cheaper" these days.

      1. Alan Brown Silver badge

        Re: Axiom vs Meme

        "Looking at IBM's general performance"

        Nothing new there. A long time ago an IBM mainframe we had (on full maintenance) failed. Phone calls to the 24 hour helpline rang out over a 3 hour period and at 5pm switched to a message saying they were closed for the day.

  7. Gene Cash Silver badge

    Stress tests

    Sometimes the weirdest shit is more stressful than "yank the power"

    I have a Linksys WRT1200ACv2. It has no problem with "yank the power"

    Pull out the WAN cable, however, and it loses its shit. It changes its IP to a random 192.168.x.x or 10.0.x.x subnet, loses all the DHCP reservations, and loses all the opened ports in the firewall. The only way to find it on the net is to do a DCHP request and find who answers.

    Very annoying and I only discovered this long past the time I could return it to Best Buy. Installing openWRT fixed it though.

  8. Anonymous Coward
    Anonymous Coward

    The majority of actual IT function for IBM Australia is provided by IBM India. It was likely a calculated risk, locking out non-Australians would knowingly impact their offshore operations, but having no resources locally to perform admin work anyway, it was something they just had to accept.

    Unfortunately I've been involved in more than one "The running config wasn't saved to the permanent config" while working at IBM, it's something all network operations people should be entirely aware of by now, and is common enough and avoidable enough to be entirely inexcusable.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon