back to article Bezos DDoS'd: Amazon Web Services' DNS systems knackered by hours-long cyber-attack

Parts of Amazon Web Services were effectively shoved off the internet today – at times breaking some customers' websites – after the cloud giant came under attack. Unlucky netizens were intermittently unable to reach sites and other online services relying on the internet goliath's technology as a result of the ongoing outage …

  1. Jan 0 Silver badge

    Back in pre-SPARC days a Sun workstation could cripple any Mac on the Internet with a "spray" command. Is this really progress and shouldn't we have fixed it by now?

  2. steviebuk Silver badge

    But..but..but..

    ...its the Cloud, this can never happen so we're told.

  3. Anonymous Coward
    Anonymous Coward

    Its payback by Oracle because Amazon moved off the Oracle databases to AWS.

    1. Mark 85
      Joke

      Could be but that's just a likely as it's some script kiddie upset he didn't get his tat from Amazon.

  4. swm

    Maybe that's why my bluehost website had intermittent DNS failures.

  5. john.jones.name
    Mushroom

    no dns security this is what happens

    if salesforce can manage to sort out DNSSEC why cant AWS ?

    1. Anonymous Coward
      Anonymous Coward

      Re: no dns security this is what happens

      And adding cryptographic authentication magically fixes DDoS floods how, exactly?

      DNSSEC solves a different problem, NOT this one.

      1. Giovani Tapini
        Coat

        Re: no dns security this is what happens

        Indeed it adds additional compute to servicing the request, discarded or not. Potentially making the attack more effective :)

        Flees, the forum quickly...

        1. Korev Silver badge
          Gimp

          Re: no dns security this is what happens

          And best of all, you may even get to pay for the extra compute required.

        2. Anonymous Coward
          Anonymous Coward

          flees indeed

          DNSSEC is NOT the problem in fact its some of the answer...

          when talking about reflection-amplification DDoS attacks, we need to look at the real culprits:

          o Systems generating traffic with spoofed IP addresses and networks allowing such traffic. This is, in fact, a root cause of reflection attacks, although the one that is nearly impossible to track back currently. Many such systems together can be operated as a botnet.

          o Reflectors and Amplifiers: remote applications that will respond to requests coming from compromised hosts. These responses will be directed at the target specified by the spoofed source IP address in the requests.

          basically AWS need RPKI and DNSSEC implemented NOW but so much for the marketing dept taking that onboard... we need someone who cares and has the technical ability to actually champion it...

          1. P. Lee

            Re: flees indeed

            Switching to TCP would fix the spoofing issue. Amplification is still an issue but at least we'd get some real IP addresses out of it to track down the botnets, making every attacking host reveal itself.

            DNSSEC would be nice, but that fixes a different problem and I think I prefer DNS over TCP - or rather TLS, which is an easier thing to do.

            1. vgrig_us

              Re: flees indeed

              And what are you gonna do with those "real ip addresses"? Notify ip cam owners that they've been hacked? :-)

              And for that you suffer considerable overhead of tcp.

              No, the answer is in the article - local caching dns resolver. Takes 5 minutes to install.

              1. diodesign (Written by Reg staff) Silver badge

                "the answer is in the article"

                There was a reference to DNS caching in the article, though I took it out as it was potentially confusing: a reader privately pointed out to me that the TTL of .amazonaws.com domains is in the order of a few seconds, much shorter than I assumed. Next time I'll check the TTL...

                So if you are caching the queries somewhere, great, but ensure they are cached for a while and not dumped within a minute.

                C.

                Edit: I've added some brief advice to the update in case it's needed by anyone in future.

                1. rcxb Silver badge

                  Re: "the answer is in the article"

                  the TTL of .amazonaws.com domains is in the order of a few seconds

                  You can override the minimum TTL in many DNS servers. A few will continue to serve up expired data from the cache if it can't reach the live server, or will cache everything to disk where you could do lookups or rig up a local authority to serve it up until things get resolved.

                  1. diodesign (Written by Reg staff) Silver badge

                    DNS caching

                    Yeah, there are various ways to solve it.

                    C.

                2. vgrig_us

                  Re: "the answer is in the article"

                  @diodesing and people say our company's 1 min ttl is bad netizenship!

                  Anyway - that's the first thing Amazon should have changed when under attack.

                  Also read somewhere that most recursive resolvers don't understand ttl below 30 sec.

          2. buchan

            Re: flees indeed

            > basically AWS need RPKI and DNSSEC implemented NOW but so much for the marketing dept taking that onboard... we need someone who cares and has the technical ability to actually champion it...

            DNSSEC has nothing to do with this.

            For RPKI to help here, it would require the entire internet to support and enforce RPKI, so I don't know whose marketing dept you are talking about ...

    2. Anonymous Coward
      Anonymous Coward

      Re: no dns security this is what happens

      because it's such a BIND

      1. Mr.Nobody

        Re: no dns security this is what happens

        There need to be third and fourth options for upvoting based on terrible humor.

        1. Anonymous Coward
          Anonymous Coward

          Re: no dns security this is what happens

          I'll look into it and get back to you

          1. Anonymous Coward
            Anonymous Coward

            Re: no dns security this is what happens

            oh lord was that a recursive DNS joke?

    3. Anonymous Coward
      Anonymous Coward

      Re: no dns security this is what happens

      DNSSEC (or rather AWS bodging their mitigations) did cause additional problems for ISPs' validating resolvers.

      It would not have prevented the DDoS.

  6. Dinanziame Silver badge
    Paris Hilton

    Wow. If Amazon is anywhere near as competent as they should be, this means somebody really heavy is after them. I suspect that their standard loads would crash most systems already. Did they piss off a country recently?

    1. iowe_iowe

      Just wait till the gloves come off, and the national players really show how they can mess up comms and infrastructure.

    2. Sgt_Oddball
      Holmes

      It could also be...

      Some very creative types seeing just what it takes to do damage to AWS. This could just be an opening salvo before they make these attacks wider reaching.

      It could also be they're seeing if/how it affected certain clients like cloudflare.

      Feels like a probing attack to me.

    3. Sil

      Perhaps they've become complacent and aren't as competent as they should be.

    4. rcxb Silver badge

      More likely they're growing too fast and haven't paid enough attention to find their own weak points. Besides, AWS isn't a premium service... they only have a huge customer base because they keep things cheap. Reliability is expensive.

    5. Anonymous Coward
      Anonymous Coward

      I wonder how much you'd have to pay amazon for compute and network to generate enough traffic to take down amazon? And how long it would take them to realize the attack CAME FROM INSIDE THE HOUSE!

    6. Happytodiscuss

      Well, there is the matter of Bezos versus Trump

      I am not saying that the US participated in this DDoS but that The US DOD just chose (Thursday) Azure after a review of the original Amazon award supposedly initiated by the white house.

      The idea of a state level actor affecting what we assume is the most durable and reliable Cloud service peopled by world class experts, does make one wonder about what the heck is going on, and if a Trump 'friend' is amplifying the big man's malevolent wishes for Bezos.

      Yeah more of this stuff coming I'm afraid.

  7. TheProf

    Longer

    Well that explains why the light on my connected to the internet smart socket* has been flashing randomly today. But that was happening on Saturday night and parts of Sunday too. Has this attack been going on for longer?

    *Yes I know but it's only being used to turn on a small LED lamp. And half the time I use the button on the device itself. Look just stop judging me.

    1. Hans 1
      Thumb Up

      Re: Longer

      Yeah, maybe it was part of the botnet that did the DDoS'ing.

  8. Marco van de Voort

    Maybe they should use a cloud

    Cloud services advertise with always up.

  9. FuzzyWuzzys
    Facepalm

    Hooray for cloud-hosting

    Lots of thrid-party providers rely on AWS, my Wordpress hosting service hosts it's infrastructure on AWS and the services have been up and down like a fart in a cullender that doesn't know which hole to escape through!

  10. Crazy Operations Guy

    "mybucket.s3.amazonaws.com"

    Storage dependent on a high-level protocol to function? That's just asking for trouble... Well, so is relying on DNS TTL-abuse to do the work of redirecting clients when a storage endpoint moves rather than doing the work of writing the protocol to handle that sort of thing properly.

    By properly, I mean something like the client connects to a 'storage Director" service, sets up a persistent connection and requests the location of a specific storage bucket along with a flag of "Inform me if the location of this bucket changes". That way the client isn't hitting the server every few second to re-resolve the bucket's location. At the very least, a DNS outage would only cause failure of any new client connecting to the director system, but already connected clients wouldn't have a problem.

    1. buchan

      Re: "mybucket.s3.amazonaws.com"

      > By properly, I mean something like the client connects to a 'storage Director" service, sets up a persistent connection and requests the location of a specific storage bucket along with a flag of "Inform me if the location of this bucket changes".

      And when your "storage Director" service is taking millions of TPS, how do you scale it?

      When it is taking millions of concurrent TCP connections, how do you scale up to handle more connections (taking into account that you only have ~64k TCP ports available per IP address)?

  11. Anonymous Coward
    Anonymous Coward

    uptick in tcp syn attacks

    Noticable uptick in tcp syn attacks recently and this one seems cleverer than most as its very low level from a wide range of (probably fake) ip addresses. So only 10-20 from each ip in an hour but from a large number of ip addresses from different subnets. Unless you are monitoring the number of connections in a TCP_SYN state you probably won't even notice it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like