back to article IT Pro confession: How I helped in the BIGGEST DDoS OF ALL TIME

I contributed to the massive DDoS attack against Spamhaus. What flowed through my network wasn't huge - it averaged 500Kbit/sec – but it contributed. This occurred because I made a simple configuration error when setting up a DNS server; it's fixed now, so let's do an autopsy. The problem I should start off by apologizing to …

COMMENTS

This topic is closed for new posts.
  1. DS 1

    You are right.. and wrong

    Firstly, brave of you to actually outline your edge of network box and its somewhat out of date configs - thus making it a target - for this - 50 lashes :)

    DNSSEC and DNS work require skills and training, things businesses of many a note cease to provide. They have reached a point where like everything else, they can't be bothered to do right.

    And 50 lashes for you claiming guilt. You did not start this attack, nor did you ask for it. There are however people who are responsible, and they are not being dealt with. Put the blame where it really lies. Cyberbunker and its miscreants.

    1. LarsG

      Re: You are right.. and wrong

      The Biggest DDOS attack...... Until the next time

    2. Trevor_Pott Gold badge

      Re: You are right.. and wrong

      Network edge is getting upgraded. Have you seen these? I think I'm in love.

      1. Aitor 1

        Re: You are right.. and wrong

        Quite nice low power sever, still I find it a bit expensive (<1000$), but my guess is that you will use it non stop for 6 years.. and use the old one as backup , right?

        1. Trevor_Pott Gold badge

          Re: You are right.. and wrong

          Yeah. Actually, the "old one" is actually 2 Atoms. (The primary and the cold spare.) So I would have 2 spares on the shelf to back up the shiny. That said, it would cost far more if I were to try doing the exact same thing but with CISCO on the box...

          1. leexgx
            Linux

            Re: You are right.. and wrong

            well the good thing is you at least had the sense to email you once your DNS was under attack/high load was happening, most who setup and DNS and forget about it, most do not even notice it as they have high speed connections normally

            most other DNS providers just send an angry email when they are sending out large amounts of data when in fact they should be fixing there own DNS to stop it from doing that (cloudflare has had lots of angry emails when an DNS amplification attack is going on, when it is there Own fault for setting up the DNS incorrectly)

          2. Fatman

            Re: with CISCO on the box...

            You know someone has to pay for those executive bonuses and excessive marketing costs, don't you????

      2. MondoMan
        Unhappy

        Re: new edge not so much...

        It's only available in black??? You would think all those gamers who grew up to be sysadmins would have forced the industry to offer some nice UV fluoro-bling!

      3. Mike Pellatt
        Thumb Up

        Re: You are right.. and wrong

        I've been in love with Supermicro for ages :-) They seem to get server design Right, especially mid-range, unlike Dell (wot, only 2.5" drives in your 2U server ?? I'm not made of money)

    3. robin48gx
      Meh

      Re: You are right.. and wrong

      Ah I thought DNS worked by looking at /etc/hosts and then always doing a recursive if it was not in there.

      The plot thickens, thanks for explaining this point.

  2. This post has been deleted by its author

    1. Professor Clifton Shallot

      Re: interesting

      Yes, more of this sort of thing, please, Reg.

  3. Anonymous Coward
    FAIL

    You're lucky...

    That your servers didn't reside on networks from TransIP, LeaseWeb (Dutch hosting providers) or even GoDaddy. Because setting up a recursive DNS server completely open for all to (ab)use gives them the right to terminate your connection entirely until you "fix your shit", and to my knowledge they will exercise that right too.

    DNS administration isn't for everyone. No offense but there are some mistakes you shouldn't make, this only tells me that you didn't go over your entire setup as carefully as you should have. Also makes me wonder if you really take (/ took?) DNS as seriously as you should.

    Because that's the problem with the Net these days: ignorance. "Nah, I'll fix that "tomorrow" because there's no way they'll abuse a server as small as mine".

    bzzzzt.

    1. Tom 38

      Re: You're lucky...

      Yep, top fail. You should never run an authoritative DNS server as a cache, you should run separate instances of them on different interfaces if you require both DNS caching/recursive lookup services internally and authoritative DNS externally.

      If DNS isn't your main job, you might look at easier to use alternatives to BIND. BIND is really powerful, but some of that power is the ability to shoot yourself in the foot. Something like djbdns is much more thought out and less error prone for the novice than BIND.

      1. Anonymous Coward
        Anonymous Coward

        Re: You're lucky...

        I would highly recommend using DJBDNS over BIND.

        For a start, the tinydns server itself, apart from being extremely fast and low impact on the server, doesn't respond to recursive queries, and doesn't have any of the numerous bugs and flaws that BIND has. I believe Dan Berstein is still waiting to pay out on his promised reward to the first person to report a verified security hole - http://cr.yp.to/djbdns/guarantee.html.

        Secondly, the function of recursive lookups is provided by dnscache, a companion daemon which you only expose to your internal network. I use addresses in the private range, and have it listening only on an internal NIC.

        1. Destroy All Monsters Silver badge
          Holmes

          Re: You're lucky...

          Oh yes. After BIND turned out to be an incomprehensible mess that moreover opened up the server to hacks back in the '00s, I turned to djbdns. Easy, comprehensible, straightforward. It had some bizarre quirks and illiberal behaviour due to the grammar nazi attitude of Bernstein which earned me complains from the front office in some rare cases.

          Though patching it at each install to make it work with modern compilers was a pain. Way to go putting up that kind retarded license prohibiting third-party distros, DJB.

          I hear licensing has been fixed in the meantime and improvements are rolling out but, well, I'm using PowerDNS now. Recommended and you can inject LUA scripts.

          1. PyLETS

            DJBDNS now public domain.

            Since 2007. And nicely supported in Debian package format. Had to compile it myself before then.

            https://en.wikipedia.org/wiki/Djbdns

  4. ukgnome

    This sort of article is why I continue my readership of the Reg.

    A very informative article.

  5. Alex Brett

    As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target. As the request is quite small it isn't too difficult to send lots of them, and by targetting the request appropriately you can get the response to be quite large, thus causing the DDOS.

    This also means that often simply turning recursion off in BIND is not sufficient, as in the default configuration it will likely (depending on which version of BIND) still return the list of root servers as a referral instead of simply refusing the query. The list of root servers is quite a large response on its own, and thus can be used in this attack.

    The magic line you need to add is "additional-from-cache no;" - this will stop that behaviour.

    1. pixl97

      Yes, it was an amplification attack. The attacker sends a small packet, bytes generally, the server replies with a larger packet. 512 bytes with the old behavior and much larger packets with the large udp packet behavior. Any request that sends back more data then send to the wrong host can be described as an amplification attack. It's the magnitude of the DNS response that makes it so effective.

      You are right on BIND, the correct response is not to return large amounts of data if you don't have the answer.

    2. PyLETS
      Boffin

      @Alex Brett

      "As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target."

      This aspect of the attack would be largely defeated by BCP 38, also known as RFC 2827 . Basically, spoofed UDP packets pretending to be from the party being attacked would not get through an ISPs customer-facing routers, unless sent from the same network as the party being attacked.

      Sure, DNS configurations also need tightening up here as Trevor and some of the comments describe, but this is also something the ISPs need to get to grips with, and start offering worse peering terms to those which don't .

      1. Oninoshiko
        Megaphone

        Re: @PyLETS

        While that's great IN THEORY, and I would love for ISPs to jump on this, as a DNS operator you can't depend on the ISP to do this. We should be checking for this kind of thing as early as possible, yes, but we also need to sanity check at later points for casses where some don't get it right or don't care.

        Are the ISPs part of the problem, yes, but Trevor was not part of the solution (until recently). Hats off to maning up about it, and solving it.

      2. Anonymous Coward
        Anonymous Coward

        Re: @Alex Brett

        I wholeheartedly agree.

        There's no reason that I should have to block inbound traffic from 172.16/12, 10/8, 169.254/16, or 192.168/16 on the outside interface of my firewall, but I do.

  6. IT Hack

    Well done for not putting your head in the sand and actually doing a post mortem and actually publishing it. Kudos.

  7. koolholio
    WTF?

    All relational to root

    DNS amplification Is more to do with forwarding queries to root servers than recursive lookups, if you wish to have a look through some whitepapers, although recursive lookups can play a part of it, hopefully you enabled DNS spoofing/DNS cache pollution protection too and spool size to limit queries to an adjustable level?

    Rate limitation only works so far given it can be small but many... the counter to this is big but few

    What about 3 Mobile's dns resolvers that queries/matches the ip listed in the SOA record and checks that against the A record... and will fail to resolve if one cannot be resolved successfully... as is, with section 2.4.2 of here: http://www.iana.org/procedures/nameserver-requirements.html

  8. Christoph
    1. Gordon Fecyk
      Thumb Down

      Ugh, "Cybergeddon."

      Just the link makes me cringe in disgust: "The great DDoS Cybergeddon of 2013." We're still here, aren't we? No one was hurt, right? Maybe spam increased for a few nanoseconds while Spamhaus was unreachable for a short time? Do I need to warm up my snow blower to deal with the deluge of junk e-mail?

      Seeing as I'm posting this to a Europe-connected network that was supposedly strongly impacted by this, I'd say this was a storm in a teapot. But SANS has a history of sensationalizing internet events. They want to be the weather.com of cyberspace and it's embarrassing.

      1. GregC

        Re: Ugh, "Cybergeddon."

        Did you actually read the article? They make it very clear early on that:

        "Here is what did not happen: the Internet did not come close to coming down, not much real impact was felt outside the victims and those in close Internet-proximity to them and we were all still able to get to pinterest and see cat pictures online."

        I got the distinct impression that the title was meant to be ironic, the piece itself was (like Trevor's, I should say) both informative and interesting, especially to someone who doesn't know the subject well (ie, me!)

        1. Trevor_Pott Gold badge
          Pint

          @GregC

          If my blitherings are interesting then I fear a walkabout outside where the daystar is might be advisable. It seems you need some of those photons that the great big ball of fusion in the sky spits out to help you create some vitamin D and jumpstart the "removing crazy" subsystems. That or oh look it's beer o'clock on a long weekend, bye!

          1. GregC

            Re: @GregC

            No need to worry on my account, all things in moderation! I'm now geeked out for the week and will indeed be enjoying the sunshine at the weekend.

            Nope, hang on. I suspect I'll be frozen solid by the time Sunday's done...

            1. Trevor_Pott Gold badge
              Pint

              Re: @GregC

              So long as you have a good weekend, sir, then all is good. Cheers and beers!

        2. Gordon Fecyk
          Thumb Down

          Then SANS needs to stop cheapening Armageddon. And Pearl Harbor. And September 11th.

          Catch phrases that invoke disgust:

          "Digital Pearl Harbor."

          "Digital Armageddon."

          "Digital 9/11."

          No one died due to any event reported using these catch phrases. Real people died in the real events. That is my problem: Cheapening the memory of real life events that killed real people.

          If they want to stop disgusting me, SANS needs to stop using them.

          1. Destroy All Monsters Silver badge
            Big Brother

            I shall have to demoralize you...

            Unless you are a born-again X-tian, Armageddon is not a "real event".

            You may also notice that "Pearl Harbor" and "9/11" were attacks on Amurrica used at full efficiency to rally the rubes behind fascistic agendas of full-scale domination, so in a sense "digitalizing" them is fully in-line with we are looking at here.

            You are also missing "Digital Attack on the Maine" and "Digital Gulf of Tonkin".

            1. Trevor_Pott Gold badge
              Pint

              Re: I shall have to demoralize you...

              Far more tragic would be the digital death of our beloved Playmonaut; a tragedy of intertubes proportions which would cause Register readers to rise up in droves against the evil aggressor.

              R.I.P. Playmonaut.

  9. joeW
    Thumb Up

    A++ read

    It's worth 50 bits of click-bait about Stephen Fry or Freetards.

  10. The Vociferous Time Waster

    Wait...

    You put all your traffic to your 'datacenter' through *one* server? I bet you run SBS as well.

    1. Trevor_Pott Gold badge

      Re: Wait...

      There are redundancies. It's actually a cluster of 2 devices. I didn't really want to get that deep into it though. I wanted to talk about the DNS not setting up a cluster in CentOS.

  11. John Deeb
    Boffin

    never "forget" any edge system!

    Since my first time as victim of a hack was because of out-to-date BIND somewhere later in the nineties when I was really just a junior apprentice admin, I'd never "forget" what was running at the edge somewhere. It was Christmas morning though and I had to try to investigate from home using shaky ISDN dialup, no physical transport being available and trying to remember various IP addresses since all DNS was down. But the lesson I learned there was that the first machines I'd upgrade or keep at least to a recent level would be the ones opened up to the Interwebs like that. That's the only flaw I could find in the article's analysis, to have the priorities really ass backwards as it was stated that almost everything was gone over to CentOS 6 apart from one of the main attack vectors in the network! Not knowing or investigating BIND's defaults seems nearly irrelevant (although might have helped). Better have a proper upgrade policy and inventory in place. That's where proper management and security starts but definitely will not end. But the scrubber might have seen like a hassle to upgrade?

    1. Trevor_Pott Gold badge

      Re: never "forget" any edge system!

      RHEL 5 is still under active support. There's nothing wrong with using it in live production. The system is updated religiously. The hardware refresh would have taken it to 6, but that got knocked back by about 6 months into April.

      So while I may have forgotten which OS the little blighter was running (my mental filing system had ticked it over to "CentOS 6" already) it was still running a maintained, patched, and secure OS.

      1. foo_bar_baz
        Linux

        Re: never "forget" any edge system!

        RHEL (and CentOS) are meant to be used in enterprise environments where things don't change quickly. This means Red Hat maintains an "old" version of a software component way after others have upgraded, religiously backporting patches to make sure it's secure. In other words stuff like BIND and openssh will have ancient version numbers, but still be up to date.

        If you want cutting edge, either compile the software yourself or use another distro. I like CentOS myself.

  12. Gordon Fecyk
    Go

    Here's a workaround on Windows Server

    This was unexpected; Thanks for coming forward. It made me brave enough to come forward with a similar workaround I did on my Windows Server setup.

    I publish a small handful of domains on that HP Microserver I bragged about a few weeks ago. One of those domains is an Active Directory domain as well as a publicly visible domain, and they handle internet e-mail and other internet things, so the domain controllers and dependent servers need to do recursive queries on the DCs running DNS, as well as host the DNS zones that make AD possible.

    Windows doesn't have an IP access list saying who can do recursive queries and who cannot. But nothing stops you from copying the zones to another non-DC DNS server and disabling recursion on that.

    I already have a reverse proxy server for various things; I just added DNS to it and port-forwarded DNS connections to it instead of to one of the DCs. Then I set up secondary copies of all my zones on it, and disabled recursion. I now have an edge DNS server that doesn't allow recursive queries and still acts authoritatively for my zones. I can still permit zone transfers from it to authorized servers outside as well, and do notifications of zone changes.

    Sure, this is, 'duh, captain obvious' stuff for some. Who would have thought DNS would be used as a DDoS vector though? If Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott can come forward with this, so can I.

    1. Trevor_Pott Gold badge

      Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

      Surely I'm not that negative! I like stuff that actually works just fine. I get tetchy when it doesn't do what it is supposed to. Or costs more than having a human do the same job. Or I haven't had coffee...

      1. Gordon Fecyk

        Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

        That was the impression I got here and here. Who in these forums brought up the impact of first impressions?

        1. Trevor_Pott Gold badge

          Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

          I calls things like I sees 'em. Good or bad. I don't hate any technology - except the things that let you robo call people in the middle of the night - but I do hate it when technology is badly implemented. A great example is saying "Trevor hates Windows.' I don't. Not even a little.

          Oh, I hate lots of things about how Windows 8 and Server 2012 have been handled, but this doesn't mean I hate all of those operating systems. Nor does it mean I think they can never be made to not suck in future releases. I fact I have litterally begged Microsoft to make the relevant changes...because I think Windows is a useful tool.

          I hate Microsoft's licensing department. I love Microsoft's storage team. I hate some of the very strict rules that Spiceworks has surrounding community interaction, but I love the opportunity it presents me to interact with other sysadmins and vendors.

          You really, really, have to do an awful lot to get on my personal hate list. Even Oracle isn't on my "no buy, ever" list. And they take hostages! Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.

          Technology is a tool. Corporations are groups of people each with individual hopes, dreams, goals and ideals. If technology sucks then I'll pan it. If it's great, I'll praise it. If it's boring, I probably won't even write about it. I'm harsh. I'm honest. I'm as up front as I know how to be. That's my job after all...

          ...biting the hand that feeds IT.

          1. Fatman

            Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

            Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.

            You can add me to the club of people who have a severe disdain for anything SONY for exactly the same reason (a rootkit infection that killed a system).

    2. koolholio

      Re: Here's a workaround on Windows Server

      http://support.microsoft.com/kb/198408 <<< theres some hidden gems within!

    3. Anonymous Coward
      Anonymous Coward

      Re: Here's a workaround on Windows Server

      If you just "need to do recursive queries" for "internet e-mail and other internet things", then you should consider removing all the root servers from your root hint list, and just using your (local and) upstream provider(s).

      This doesn't make the problem go away -- you can still get rogue external requests -- put it translates them into a problem for your upstream supplier. In a DDOS situation, (1) your upstream supplier will have (hopefully) respond from cache, and (hopefully) have better quallified 24 hr support, who can pull the plug.

  13. Colin Miller

    filter on sane sender address?

    Do ISP filter *outbound* traffic to ensure that the sender's address is sane?

    The filtering could be done around the NAS level, or at the peer-exchange (LINX).

    1. Oninoshiko
      Boffin

      Re: filter on sane sender address?

      Most, no.

      should they? yes.

  14. J.G.Harston Silver badge

    Kessel Run?

    "it averaged 500Kbit/ses"

    For how long? That's a speed, not a volume.

    1. Trevor_Pott Gold badge

      Re: Kessel Run?

      13 hours and change. In my defence, I was asleep for most of it...

  15. Anonymous Coward
    Anonymous Coward

    "The keen eye will notice two other flaws in my server design. The first is that BIND isn't chrooted. This is because the spywaredomains.zones file from malwaredomains isn't really designed with RedHat-based operating distros in mind. If you were to chroot bind you'd have to post-process the zone file to cope with the path differences."

    The paths are relative to the chroot, so say you chroot in /var/named, you could just copy the blockeddomain.hosts file to /var/named/etc/namedb/blockeddomain.hosts. No post-processing needed. Shared virtual hosting and fail2ban have nothing to do with this - chrooting BIND is there to make it harder to exploit bugs in BIND.

    "The second is that DNSSEC isn't enabled."

    One of the reasons why this is such an efficient traffic amplifier is that a DNSSEC signed zone can have a much larger response to a (small) query than an unsigned zone. Not saying that it isn't useful, but DNSSEC does require care, really wants modern software probably with rate limits (which are non-default build options / patches in most current implementations), and keeping track of development. There are exciting new opportunities to break your DNS with it too, of course.

    Since there were a few comment suggesting djbdns for inexperienced admins - oh $DEITY no, please....

    1. Trevor_Pott Gold badge

      The particular implementation of BIND + chroot utterly refused to look in the chroot directory for /etc/namedb, no matter how much tinkering I tried. I gave up eventually and left it. As for the shared virtual hosting and fail2ban comment, that is there because most of the "bugs in BIND" we might care about are exploits that work if you have manged to gain a remote console.

      SSH on an alternate port + fail2ban + not actually giving the information to anyone and having a very small user footprint means your chances of getting into the system to exploit BIND in that fashion are hella slim. There is always the remote possibility that you could use some sort of remote attack against BIND like that, but the chances are even smaller. In terms of the risk posed, I think I can get away with not chrooting the thing for the 2-3 moths between initial roll out of the service and the replacement of the unit with a CentOS6 box.

      At least on CentOS6 the bloody chroot works right and the malwaredomains zone works without post-processing the text file. I should also point out that the DNSSEC implementation set up in CentOS6 is actually pretty good.

  16. Anonymous Coward
    Anonymous Coward

    "edge scrubber"?

    I almost stopped reading there, but you lost me at honeypot. As far as I know, a honeypot is a place where you profile/catch attackers. Wikipedia says:

    "In computer terminology, a honeypot is a trap set to detect, deflect, or in some manner counteract attempts at unauthorized use of information systems. Generally it consists of a computer, data, or a network site that appears to be part of a network, but is actually isolated and monitored, and which seems to contain information or a resource of value to attackers."

    I don't think honeypot is the correct term to use when setting up a machine for the security of your own users. I like your article Trevor, regardless of your DNS "sin" - yes, it really is that bad. However, as an tech author, you should really use common place terminology. Since I'm not in the UK, I could be wrong, and that is how IT professionals talk over there.

    1. Trevor_Pott Gold badge

      Re: "edge scrubber"?

      Yes. A honeypot is indeed where you profile and catch attackers. Why are you hitting the honeypot machine if you aren't clicking on stupid things or are an attacker? They honeypot allows me to catch not only attackers but stupid users. I would say that "redirecting a user to a honeypot machine that displays an error or educational message when they try visiting a site on the list, then logs the thing so I can find and LART someone" counts as a honeypot.

      As for edge scrubber, the system also does IDS and DPS. It scrubs my datastream. It leaves on the edge of my network. What the hell would you call it?

      If it's a ship and it goes through the gate, you call it a gateship. You only call it a puddle jumper if you need something that sounds good on TV. It's an edge device, it scrubs my datatream. Should I call it a boysenberry?

      1. Adam JC
        Thumb Up

        Re: "edge scrubber"?

        Upvote for the 'puddle jumper' reference. :-)

        1. Fatman

          Re: "edge scrubber"?

          Add one more for me, too!

      2. Fatman

        Re: ...if you aren't clicking on stupid things

        That is what WROK PALCE does to prevent damagement from viewing pr0n sites.

        Boy, did some male execs scream when the female CEO got an email listing their attempt in real time to access a prohibited site. She often paid a visit to their office - unannounced!!!!

        IT got called some really nasty names; but, what the hell, we were only following ORDERS!!!!!

  17. Steve Knox

    Caching?

    If someone asks your server "where is www.google.com" a whole bunch of times then your server starts flooding google.com's DNS servers.

    Not if your server is set up to cache recursive results for a period of time (which I believe is the default.) More likely attackers are asking for:

    www1.google.com

    www2.google.com

    www3.google.com

    .....

    which would result in multiple lookups even if caching is enabled.

    Feel free to correct me if I'm wrong.

    1. Trevor_Pott Gold badge

      Re: Caching?

      Nope, you are 100% correct. If you are attacking properly that is exactly how you do it. (Actually, it is is the DNS for www.google.com you want to take down you attack with 1.www.google.com and 2.www.google.com etc.) That said, I was a little out in the weeds on describing the attack as is, and the sysadmin blogs are supposed to be 600 words. Had to leave out some details somewhere. :)

      1. koolholio
        Boffin

        Re: Caching?

        *interjects* You would need a method of applying an address answer limit... but then surely this could also be covered by:

        http://tools.ietf.org/html/rfc2827 or http://tools.ietf.org/html/bcp38

        it says primarily about forged packets, I assume that would be dns spoofing or even related to cache poisoning? Is there a difference between the two?

        1. Trevor_Pott Gold badge

          Re: Caching?

          Network ingress filtering requires you be "part" of the wider internet, rather than merely the equivalent of a consumer with a fat pipe. We don't have access to BGP. We have no way of seeing, processing or acting upon the internet's wider routing table. Without this, the sort of ingress filtering duscussed in those documents simply isn't possible.

          So what's left? Whitlisting systems manually that you want to connect to your DNS in iptables? How's that work when some of those units are mobile? Users with dynamic residential IPs, connecting from hotels or even over mobile links? What we really need is a DNS server and client infrastructure that allows for authentication of clients before they can look things up. DNS + TLS if you will. It might be time to start building something internally similar to opendns' infrastructure. I'll give it a thought.

          1. koolholio
            Boffin

            Re: Caching?

            Or, depending upon your network setup... you could implement the use of a router/switches iptables/netfilter (provided it has --match --hexstring and --algo filters) by matching the request for the recursive flag set on usually UDP packets inbound at a certain offset. I believe iptables/netfilter is included within most linux and unix distros. Zeroshell (a linux based router distro) may even allow you to enter raw commands to utilise this.

            Wireshark is useful for finding what offset and the dns query flags--- which is the hex string you wish to filter for... you may also apply a rate limiter using the same patterns, but with the rate respectively.

    2. Alan Thompson
      Stop

      Re: Caching?

      Not quite right -

      ----snip----

      If someone asks your server "where is www.google.com" a whole bunch of times then your server starts flooding google.com's DNS servers.

      ----snip----

      The correct statement is:

      If someone asks your server "where is www.google.com" a whole bunch of times while spoofing a source address of [one of spamhaus's external IPs] then your server starts flooding spamhaus's external IP address with large DNS replies. Local caching means nothing.

      Then spamhaus blacklists your IP address

      Then all of your email firewall's requests to spamhaus start being blocked

      Then you can't evaluate incoming email traffic against spamhaus' database

      Then you start letting spam in.

      THEN since this is a DDOS attack from many improperly configured DNS servers, spamhaus' servers go offline.

      This is a DNS amplification attack because small amounts of DNS specific traffic from one group of attackers to a single DNS server results in large amounts of traffic to the victim.

      1. Trevor_Pott Gold badge

        Re: Caching?

        That is one potential variant of the attack, yes. It is not the only one. There are a few others too. Oh DDOSes, so many of you out there!

      2. This post has been deleted by its author

      3. koolholio
        WTF?

        Re: Caching?

        so are you saying that TTL, expiry and any cache including an EDNS0 cache timeout are redundant and are of no effect in relation to caches and if that is the case... caches may aswell not exist...

        If that is the case, I also think a cached response shouldn't have its own flag assigned to it?

  18. Sixtysix
    Pint

    Top banana

    A post that can be logged as CPD after reading.

    Win win - virtual pint on me.

    Cheers

  19. Anonymous Coward
    Anonymous Coward

    Open DNS fishing

    I noticed small bursts of outsiders attempting recursive queries from 12 March. All were rejected as 'outside'. The bursts grew in size until 23rd March then stabilised until 28th. Not much seen since.

  20. Justin Clements

    Losing Sleep?

    To be honest, I wouldn't lose sleep over accidentally running a DDoS on Spamhaus. Everyone in the industry has been frustrated by them at some point in the past, and frankly, they are pretty much getting what they deserved.

    There are plenty of other organisations who provide the same service but with less attitude.

    1. leexgx
      Happy

      Re: Losing Sleep?

      spam supporter detected

      1. leexgx
        Megaphone

        Re: Losing Sleep?

        email would be practicably be useless with out Spamhaus and the email servers that use there lists

  21. Alan Thompson
    Alert

    Publish External DNS to Your ISP - Maintain Local Control

    Whenever I set up a new network/DNS zone, one of the first things I do is to configure the external version of the zone as MASTER on the edge DNS server (similar to your scrubber). However, my ACLs prevent external access from the Internet to DNS except by my ISP's DNS servers. I then configure (or request configuration - if the ISP is still in the dark ages) the zone on the ISP DNS servers as SLAVE zones with matching SLAVE entries on my MASTER. The domain's ICANN registered servers are then configured as the ISP's DNS servers. This serves several purposes:

    1) All external DNS requests go the ISP's "properly configured", high throughput DNS servers

    2) If my edge server needs to go down for maintenance it doesn't take external DNS offline.

    3) The network admin maintains operational control of the domain and can do all the updates locally on the edge server

    4) The edge DNS server's IP address is never published as a DNS server for the domain

    5) The edge DNS server only handles zone transfers/updates to the ISP's DNS servers while maintaining its MASTER status.

    6) Edge devices on the local network can do local-external and recursive lookups on the ISP's DNS servers while internal devices use internal DNS servers (especially when using private addressing).

    I ALWAYS use a completely separate set of internal DNS servers and MASTER/SLAVE zones for internal authoritative access and recursive lookups - which also gives me the ability to blacklist bad domains there.

  22. Alan Brown Silver badge
    WTF?

    oh for fuck's sake.

    The "fix" is easy.

    in general options you set "allow query {localnets;}" (and any networks you think should be allowed to make general recursive queries)

    Then in each zone file you add "allow query {any;}"

    Porblem solved. You won't send answers for domains you're not authoritative for, except to explicitly defined networks

    It's not fucking rocket science, it's not hard and above all, it was what I was recommending 15 years ago to keep leeches off of DNS servers. You don't need DNSSEC or any of the other bullshit to reduce the nuisance factor of an open DNS server.

    Additional hacks to rate limit responses have been published. These and DNSSEC help a bit, but not as much as the simple (in most cases 1 line) config change above.

  23. Anonymous Coward
    Anonymous Coward

    Pleb

    Pleb goes in all fields

  24. taxman
    Big Brother

    And from Preventia:

    Whilst I appreciate this is a known problem, it seems to be an area of increasing risk as the attacks only in the last few days there was the largest DDos attack to date. This 300 Gbp/s attack is making ‘the internet’ shake.The internet will fail before Prolexic would (we are a virtual second internet for our customers). With 800 Gbp/s attack bandwidth available (and in the process of tripling that), Prolexic are the ONLY service/solution/product in the world that could handle an attack of this scale and duration.

  25. John Smith 19 Gold badge
    Unhappy

    I wonder if most other servers were as "badly" configured as Trevor's?

    Actual human being alerted to suspicious behavior

    Auto throttling of bandwidth cutting in even before a human response.

    I think if they had the answer would be "quite a lot better than what actually happened."

    Hopefully this will have given various sysadmins a wake up call to review their configurations and tighten up their procedures ( Unless the proverbial PHB puts their foot down and insists it cannot be changed because it would inconvenience the CEO)

    This presumes some of them even realized they were involved of course.

    1. Alan Brown Silver badge

      Re: I wonder if most other servers were as "badly" configured as Trevor's?

      Yes they are. Wide open is the default setting for Bind. Even DJB and MS wwere wide open last time I looked.

      It's the same mentality which STILL defines any DNS entries in zonefiles with zero padding as octal, despite the RFC explicitly stating that IPv4 addresses are dotted decimals. I got royally flamed when I pointed that particular "issue" out 18 years ago and asked that the RFC or the software be altered for consistency (given they were written by the same person, it didn't seem to be an unreasonable request). Not long after that, spammers started using dotted and long hex/binary/octal/decimal URLs in spam (It took filter authors to nail that down. Bind is still open to that abuse)

      1. John Smith 19 Gold badge
        Meh

        Re: I wonder if most other servers were as "badly" configured as Trevor's?

        "Yes they are. Wide open is the default setting for Bind. Even DJB and MS wwere wide open last time I looked"

        I was thinking the the human alerting of exceptional behavior and the auto throttling until the cause was investigated.

        That part of his configuration.

  26. Anonymous Coward
    Anonymous Coward

    one absolute solution

    Using openresolverprojects statistics, 25 million copies of this please:

    http://www.cloudshield.com/solutions/SP_DNS_Protection.asp

  27. koolholio
    Boffin

    Possible solutions for the opensource community

    http://www.ntop.org/products/ndpi/

  28. koolholio
    Boffin

    Possible to detect and monitor... but not so easy to filter out

    You can capture just dns requests from a dns server itself using a capture filter, such as this one:

    "<CONNECTIONTYPE> host <GATEWAYMAC> and src net <LOCALNET/CIDR> or not src net <LOCALNET/CIDR> and port 53" (optionally omitting "and udp" and changing the port if configured differently)

    of course you can specify destinations respectively, if you're doing this further upstream by using:

    host <IP> or net <IPRANGE/CIDR> or mask <netmask> if its over multiple subnets

    Which will capture all requests and responses to and from... Heres where it gets difficult:

    You would just need to apply filters to this, using pattern matching for distinguishing characteristics but there may be need for utilising comparisons within the filters.

  29. Daniel B.
    Boffin

    So I'm not alone!

    Last year I decided to switch my DNS pointers from the hosting service I have (GoDaddy) to my own. Alas, I forgot that while ns2 had the "recursion disabled by default" setting, ns1 *didn't*.

    2 weeks later, I check out my bandwidth usage and notice that it's waaay off chart. Monkeying with iptables, I was able to pinpoint the extra traffic to port 53. Firing up tcpdump gave me a zillion DNS requests for some weird domain, which itself was pointing to CloudFlare as well. Ouch! I outright blocked port 53, sending it to DROP. I even switched the DNS order ... but I did notice that ns2 didn't get the zillion requests. So I went on checking and finally found out about both the open recursion configuration, and the default config switch. Even after securing my ns1 BIND, I still had to leave port 53 blocked on my main DNS 'till the request flood died out. It cost me a lot in bandwidth that month, but lesson learned...

  30. Emo

    Scrubber

    I'd be interested in some sort of Reg Tutorial on putting something low cost together and configured, even if only for home use.

  31. Glen Turner 666

    Old, old attack.

    It's not rocket science, I described the correct configuration for AusCERT back in 1999 in response to DDoS we were seeing then. (Modify the "bogon" list for the newer "end of IPv4, so let's use every Class A possible" list of bogon networks.) See AL-1999.004 at http://www.auscert.org.au/render.html?it=80

This topic is closed for new posts.