IT Pro confession: How I helped in the BIGGEST DDoS OF ALL TIME

Top banana

A post that can be logged as CPD after reading.

Win win - virtual pint on me.

Cheers

Publish External DNS to Your ISP - Maintain Local Control

Whenever I set up a new network/DNS zone, one of the first things I do is to configure the external version of the zone as MASTER on the edge DNS server (similar to your scrubber). However, my ACLs prevent external access from the Internet to DNS except by my ISP's DNS servers. I then configure (or request configuration - if the ISP is still in the dark ages) the zone on the ISP DNS servers as SLAVE zones with matching SLAVE entries on my MASTER. The domain's ICANN registered servers are then configured as the ISP's DNS servers. This serves several purposes:

1) All external DNS requests go the ISP's "properly configured", high throughput DNS servers

2) If my edge server needs to go down for maintenance it doesn't take external DNS offline.

3) The network admin maintains operational control of the domain and can do all the updates locally on the edge server

4) The edge DNS server's IP address is never published as a DNS server for the domain

5) The edge DNS server only handles zone transfers/updates to the ISP's DNS servers while maintaining its MASTER status.

6) Edge devices on the local network can do local-external and recursive lookups on the ISP's DNS servers while internal devices use internal DNS servers (especially when using private addressing).

I ALWAYS use a completely separate set of internal DNS servers and MASTER/SLAVE zones for internal authoritative access and recursive lookups - which also gives me the ability to blacklist bad domains there.

oh for fuck's sake.

The "fix" is easy.

in general options you set "allow query {localnets;}" (and any networks you think should be allowed to make general recursive queries)

Then in each zone file you add "allow query {any;}"

Porblem solved. You won't send answers for domains you're not authoritative for, except to explicitly defined networks

It's not fucking rocket science, it's not hard and above all, it was what I was recommending 15 years ago to keep leeches off of DNS servers. You don't need DNSSEC or any of the other bullshit to reduce the nuisance factor of an open DNS server.

Additional hacks to rate limit responses have been published. These and DNSSEC help a bit, but not as much as the simple (in most cases 1 line) config change above.

And from Preventia:

Whilst I appreciate this is a known problem, it seems to be an area of increasing risk as the attacks only in the last few days there was the largest DDos attack to date. This 300 Gbp/s attack is making ‘the internet’ shake.The internet will fail before Prolexic would (we are a virtual second internet for our customers). With 800 Gbp/s attack bandwidth available (and in the process of tripling that), Prolexic are the ONLY service/solution/product in the world that could handle an attack of this scale and duration.

I wonder if most other servers were as "badly" configured as Trevor's?

Actual human being alerted to suspicious behavior

Auto throttling of bandwidth cutting in even before a human response.

I think if they had the answer would be "quite a lot better than what actually happened."

Hopefully this will have given various sysadmins a wake up call to review their configurations and tighten up their procedures ( Unless the proverbial PHB puts their foot down and insists it cannot be changed because it would inconvenience the CEO)

This presumes some of them even realized they were involved of course.

Possible solutions for the opensource community

http://www.ntop.org/products/ndpi/

Possible to detect and monitor... but not so easy to filter out

You can capture just dns requests from a dns server itself using a capture filter, such as this one:

"<CONNECTIONTYPE> host <GATEWAYMAC> and src net <LOCALNET/CIDR> or not src net <LOCALNET/CIDR> and port 53" (optionally omitting "and udp" and changing the port if configured differently)

of course you can specify destinations respectively, if you're doing this further upstream by using:

host <IP> or net <IPRANGE/CIDR> or mask <netmask> if its over multiple subnets

Which will capture all requests and responses to and from... Heres where it gets difficult:

You would just need to apply filters to this, using pattern matching for distinguishing characteristics but there may be need for utilising comparisons within the filters.

So I'm not alone!

Last year I decided to switch my DNS pointers from the hosting service I have (GoDaddy) to my own. Alas, I forgot that while ns2 had the "recursion disabled by default" setting, ns1 *didn't*.

2 weeks later, I check out my bandwidth usage and notice that it's waaay off chart. Monkeying with iptables, I was able to pinpoint the extra traffic to port 53. Firing up tcpdump gave me a zillion DNS requests for some weird domain, which itself was pointing to CloudFlare as well. Ouch! I outright blocked port 53, sending it to DROP. I even switched the DNS order ... but I did notice that ns2 didn't get the zillion requests. So I went on checking and finally found out about both the open recursion configuration, and the default config switch. Even after securing my ns1 BIND, I still had to leave port 53 blocked on my main DNS 'till the request flood died out. It cost me a lot in bandwidth that month, but lesson learned...