back to article Networking wonks say lousy planning, not DDOS, caused #Censusfail

The failure of the Australian census seems to be a failure of planning. The Federal Government is blaming a distributed denial of service attack (DDoS) and an abundance of caution for sending the once-every-five-years Antipodean citizen survey into a grinding halt beginning last night and continuing as of the time of writing …

  1. Andrew Commons

    Timeline

    The ABC has a timeline in this story:

    http://www.abc.net.au/news/2016-08-10/census-night-how-the-shambles-unfolded/7712964

    See brief extract below. Note the line that reads "Fire walls kick in". Where were they before that time? Was this an optional extra?

    Intelligence agency called in

    August 9, 2016

    11:55am

    The incident is reported to the Government's Australian Signals Directorate to seek any advice on prevention of further incidents or any intelligence-related threat.

    Fire walls kick in

    August 9, 2016

    4:58pm

    Another modest increase in traffic is automatically defended by network fire walls. "Additional measures" are taken to prevent further attempts of this type.

    1. dan1980

      Re: Timeline

      Yeah . . .

      You might also have picked-up in the press conference that the minister said that the ABS and IBM had robust DoS response plans, which they implemented after the first "frustration". (Or something to that effect.)

      It struck me as odd because DoS mitigation and protection should be bloody well in place and functioning before a service goes online - not applied as a response to an attack. When, say, a journalist goes into a war zone, they don't wait until the bullets start flying to put on a vest . . .

  2. Dramoth

    It was a DDOS attack

    By the Australian public... just trying to access the servers and fill out the damned forms.

    The good old Liberal Government went with an new fangled IT method of collecting all our data... just to prove to the Luddites that the technology is rubbish and we should move to a 10 year collecting cycle and chopping down more trees to create all those lovely paper forms.

  3. Oengus
    FAIL

    Hanlon's Razor

    Never attribute to malice that which can be adequately explained by incompetence.

    DDoS is too convenient an excuse for the outage. Any "strategic" website should have DDoS defences capable of handling all but the most determined attack and doubly so for high profile publicly funded sites. The level of DDoS attack that the Census site should have been able to defend against would be noticed by the observers.

  4. dan1980

    What really amuses me is how the message evolved as the saga dragged on.

    Yesterday evening, the message coming from the ABS was that there was no problem. There dismal of the suggestion that anything could be wrong is more interesting now that we know their problems started several hours earlier.

    Then, as the clusterf%$k became unable to deny any longer, the blame and hand-wringing started and strong words like 'hack' and 'attack' were thrown out in an attempt to convince the public that the poor ABS weren't at fault: it was a malicious, international cyber attack!

    This morning, as the evidence for such claims appeared rather difficult to come by, and those who did believe it are now worried about the security of their information, the language has changed to the point where they are grasping for any word that isn't "hack", "attack" or "breach", settling, confusingly, on "frustration".

    So, in an attempt to avoid appearing incompetent, the ABS conjured an evil international cyber-attack and then, realising that this also implied a PR disaster, these attacks evaporate into 'frustration'. The end result bing that the ABS were incompetent when it came to scaling the system and incompetent when it came to securing it. And were dishonest about both.

    Taking from the ministers penchant for repeating his key points at the press conference:

    Dishonesty and incompetence. Dishonesty and incompetence.

    1. GrapeBunch

      "There dismal of the suggestion" Just wait until I get ~my~ dismissal in their. Then yule sea sump thin.

  5. bleh_meh

    Next day still no dice

    Seems interesting to me that the site is still refusing participant access.

    You would have thought that at the end of the day after the site was shut down it would be back working in a somewhat limited fashion.

    There was reports that they pulled the plug at about 1930AEST Tuesday which should have given them a chance to let the "DDOS" (inverted commas deliberate because quotation fingers don't work online) subside but the continued downtime likely means that something is royally f**ked there!

  6. Anonymous Coward
    Anonymous Coward

    I say DOS was orchastrated by locals!

    I think this DOS was controlled by some locals. Probably angered by the longer name/address/birthdate data retention concerns.

    10am just about as early as a local attacker can be bothered to wake up. Its probably a crafted attacked not necessarily a massive DDOS as the ministers and senior ABS staff have said.

    1. Sorry that handle is already taken. Silver badge

      Re: I say DOS was orchastrated by locals!

      Several million locals did try to access the system at the same time from numerous locations.

  7. Winkypop Silver badge
    Thumb Down

    Even if it comes back up

    It has lost the crucial trust they so vigorously insisted it should have.

    Paper never refused ink.

  8. Ken Moorhouse Silver badge
    Coat

    Or how about it coinciding with someone entering the data for...

    ...someone with the name Drop Censustable

  9. mark 177
    WTF?

    DDoS?

    Apparently the site's capacity was 1,000,000 forms per hour. Isn't that a little low for a country of 15 million households who were explicitly instructed all to fill in their online forms on a single day (meaning evening for most people)?

  10. Anonymous Coward
    Anonymous Coward

    Back on line

    Well, it seems the site is back on line now on Thursday afternoon, but guess what? It is not accessible from overseas VPNs or proxies.

    What to do now?

  11. Notional Semidestructor
    Joke

    Sad but true....

    There seem to be so few in Government or among Government minions who can even craft a decent excuse nowadays, never mind doing a proper job....

    So here's a few improved excuses to help them out:

    "The dog ate my homework"

    "A big boy did it and ran away"

    "I didn't do it, nobody saw me do it, you can't prove anything"

    (sigh)

    @Oengus - is that you, Angus?

    ...we were taught the abbreviated variation: "Usually cockup rather than conspiracy"

  12. scottyman

    Appears to be blocking anyone using an overseas DNS (like Google, OpenDNS or even my own private company DNS)

    All of their content is on a CDN which is throwing a 504 error for every request, which suggests they are actually polluting the DNS cache of worldwide servers with invalid entries

    1. GrapeBunch

      fwiw, Hoxx VPN Proxy (free) reports 11 Australian servers.

  13. Anonymous South African Coward Bronze badge

    Nah, the server went walkabouts

  14. HighHair

    I prefer my own timeline...

    It's a simple case of SoftLayer not being able to keep up with user load...

    (Timeline comes from www.smh.com.au/national/census-website-attacked-by-hackers-abs-claims-20160809-gqouum.html )

    <mark>Mmmm... I’ve got some time on my coffee break…</mark>

    <b>Tuesday 10.08am:</b> Australian Bureau of Statistics online monitoring systems detected a significant increase in traffic. It lasted 11 minutes causing a system outage of five minutes. The traffic increase was suspected to be a denial of service and investigations were commenced by ABS and IBM.

    <mark>Better start it before I go to lunch…</mark>

    <b>11.46am:</b> Another increase in traffic was observed consistent with a second denial of service. A mitigation response plan to block all international traffic was activated at 11.50am, which immediately stopped the event. The ABS kept the block on all international traffic until midnight.

    <b>11.55am:</b> The incident was reported to the Australian Signals Directorate to seek advice on prevention of further incidents or any intelligence-related threat to the ABS.

    <mark>Let’s get it done BEFORE dinner…</mark>

    <b>4.58pm:</b> An increase in traffic was defended by network firewalls. Additional measures were taken to prevent further attempts of this type.

    <mark>After dinner…</mark>

    <b>6.15pm:</b> A small-scale denial of service was attempted on the ABS website and stopped by the standard denial of service protections.

    <b>7.30pm:</b> The online census form monitoring systems detected a significant denial of service. This event took a different form from those before, as there was a large increase in traffic with thousands of Australians logging on to complete their census.

    <b>7.45pm:</b> The ABS shut down the online form to protect the system from further incidents.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like