Feeds

back to article IT Pro confession: How I helped in the BIGGEST DDoS OF ALL TIME

I contributed to the massive DDoS attack against Spamhaus. What flowed through my network wasn't huge - it averaged 500Kbit/sec – but it contributed. This occurred because I made a simple configuration error when setting up a DNS server; it's fixed now, so let's do an autopsy. The problem I should start off by apologizing to …

COMMENTS

This topic is closed for new posts.

Page:

You are right.. and wrong

Firstly, brave of you to actually outline your edge of network box and its somewhat out of date configs - thus making it a target - for this - 50 lashes :)

DNSSEC and DNS work require skills and training, things businesses of many a note cease to provide. They have reached a point where like everything else, they can't be bothered to do right.

And 50 lashes for you claiming guilt. You did not start this attack, nor did you ask for it. There are however people who are responsible, and they are not being dealt with. Put the blame where it really lies. Cyberbunker and its miscreants.

21
1
Silver badge

Re: You are right.. and wrong

The Biggest DDOS attack...... Until the next time

2
2
Gold badge

Re: You are right.. and wrong

Network edge is getting upgraded. Have you seen these? I think I'm in love.

0
1
Meh

Re: You are right.. and wrong

Ah I thought DNS worked by looking at /etc/hosts and then always doing a recursive if it was not in there.

The plot thickens, thanks for explaining this point.

2
0
Bronze badge

Re: You are right.. and wrong

Quite nice low power sever, still I find it a bit expensive (<1000$), but my guess is that you will use it non stop for 6 years.. and use the old one as backup , right?

1
0
Gold badge

Re: You are right.. and wrong

Yeah. Actually, the "old one" is actually 2 Atoms. (The primary and the cold spare.) So I would have 2 spares on the shelf to back up the shiny. That said, it would cost far more if I were to try doing the exact same thing but with CISCO on the box...

2
0
Bronze badge
Unhappy

Re: new edge not so much...

It's only available in black??? You would think all those gamers who grew up to be sysadmins would have forced the industry to offer some nice UV fluoro-bling!

1
0
Thumb Up

Re: You are right.. and wrong

I've been in love with Supermicro for ages :-) They seem to get server design Right, especially mid-range, unlike Dell (wot, only 2.5" drives in your 2U server ?? I'm not made of money)

0
0
Linux

Re: You are right.. and wrong

well the good thing is you at least had the sense to email you once your DNS was under attack/high load was happening, most who setup and DNS and forget about it, most do not even notice it as they have high speed connections normally

most other DNS providers just send an angry email when they are sending out large amounts of data when in fact they should be fixing there own DNS to stop it from doing that (cloudflare has had lots of angry emails when an DNS amplification attack is going on, when it is there Own fault for setting up the DNS incorrectly)

1
0
Bronze badge

Re: with CISCO on the box...

You know someone has to pay for those executive bonuses and excessive marketing costs, don't you????

0
0
Bronze badge
Thumb Up

interesting

Very interesting article, with good, clear explanations for those of us who aren't DNS experts.

19
1
Bronze badge

Re: interesting

Yes, more of this sort of thing, please, Reg.

21
0
Silver badge
FAIL

You're lucky...

That your servers didn't reside on networks from TransIP, LeaseWeb (Dutch hosting providers) or even GoDaddy. Because setting up a recursive DNS server completely open for all to (ab)use gives them the right to terminate your connection entirely until you "fix your shit", and to my knowledge they will exercise that right too.

DNS administration isn't for everyone. No offense but there are some mistakes you shouldn't make, this only tells me that you didn't go over your entire setup as carefully as you should have. Also makes me wonder if you really take (/ took?) DNS as seriously as you should.

Because that's the problem with the Net these days: ignorance. "Nah, I'll fix that "tomorrow" because there's no way they'll abuse a server as small as mine".

bzzzzt.

6
6
Silver badge

Re: You're lucky...

Yep, top fail. You should never run an authoritative DNS server as a cache, you should run separate instances of them on different interfaces if you require both DNS caching/recursive lookup services internally and authoritative DNS externally.

If DNS isn't your main job, you might look at easier to use alternatives to BIND. BIND is really powerful, but some of that power is the ability to shoot yourself in the foot. Something like djbdns is much more thought out and less error prone for the novice than BIND.

8
0
Anonymous Coward

Re: You're lucky...

I would highly recommend using DJBDNS over BIND.

For a start, the tinydns server itself, apart from being extremely fast and low impact on the server, doesn't respond to recursive queries, and doesn't have any of the numerous bugs and flaws that BIND has. I believe Dan Berstein is still waiting to pay out on his promised reward to the first person to report a verified security hole - http://cr.yp.to/djbdns/guarantee.html.

Secondly, the function of recursive lookups is provided by dnscache, a companion daemon which you only expose to your internal network. I use addresses in the private range, and have it listening only on an internal NIC.

6
0
Silver badge
Holmes

Re: You're lucky...

Oh yes. After BIND turned out to be an incomprehensible mess that moreover opened up the server to hacks back in the '00s, I turned to djbdns. Easy, comprehensible, straightforward. It had some bizarre quirks and illiberal behaviour due to the grammar nazi attitude of Bernstein which earned me complains from the front office in some rare cases.

Though patching it at each install to make it work with modern compilers was a pain. Way to go putting up that kind retarded license prohibiting third-party distros, DJB.

I hear licensing has been fixed in the meantime and improvements are rolling out but, well, I'm using PowerDNS now. Recommended and you can inject LUA scripts.

1
0
Bronze badge

DJBDNS now public domain.

Since 2007. And nicely supported in Debian package format. Had to compile it myself before then.

https://en.wikipedia.org/wiki/Djbdns

0
0
Silver badge

This sort of article is why I continue my readership of the Reg.

A very informative article.

26
2

As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target. As the request is quite small it isn't too difficult to send lots of them, and by targetting the request appropriately you can get the response to be quite large, thus causing the DDOS.

This also means that often simply turning recursion off in BIND is not sufficient, as in the default configuration it will likely (depending on which version of BIND) still return the list of root servers as a referral instead of simply refusing the query. The list of root servers is quite a large response on its own, and thus can be used in this attack.

The magic line you need to add is "additional-from-cache no;" - this will stop that behaviour.

7
2

Yes, it was an amplification attack. The attacker sends a small packet, bytes generally, the server replies with a larger packet. 512 bytes with the old behavior and much larger packets with the large udp packet behavior. Any request that sends back more data then send to the wrong host can be described as an amplification attack. It's the magnitude of the DNS response that makes it so effective.

You are right on BIND, the correct response is not to return large amounts of data if you don't have the answer.

4
1
Bronze badge
Boffin

@Alex Brett

"As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target."

This aspect of the attack would be largely defeated by BCP 38, also known as RFC 2827 . Basically, spoofed UDP packets pretending to be from the party being attacked would not get through an ISPs customer-facing routers, unless sent from the same network as the party being attacked.

Sure, DNS configurations also need tightening up here as Trevor and some of the comments describe, but this is also something the ISPs need to get to grips with, and start offering worse peering terms to those which don't .

1
0
Silver badge
Megaphone

Re: @PyLETS

While that's great IN THEORY, and I would love for ISPs to jump on this, as a DNS operator you can't depend on the ISP to do this. We should be checking for this kind of thing as early as possible, yes, but we also need to sanity check at later points for casses where some don't get it right or don't care.

Are the ISPs part of the problem, yes, but Trevor was not part of the solution (until recently). Hats off to maning up about it, and solving it.

0
0
Bronze badge

Re: @Alex Brett

I wholeheartedly agree.

There's no reason that I should have to block inbound traffic from 172.16/12, 10/8, 169.254/16, or 192.168/16 on the outside interface of my firewall, but I do.

0
0

Well done for not putting your head in the sand and actually doing a post mortem and actually publishing it. Kudos.

20
2
WTF?

All relational to root

DNS amplification Is more to do with forwarding queries to root servers than recursive lookups, if you wish to have a look through some whitepapers, although recursive lookups can play a part of it, hopefully you enabled DNS spoofing/DNS cache pollution protection too and spool size to limit queries to an adjustable level?

Rate limitation only works so far given it can be small but many... the counter to this is big but few

What about 3 Mobile's dns resolvers that queries/matches the ip listed in the SOA record and checks that against the A record... and will fail to resolve if one cannot be resolved successfully... as is, with section 2.4.2 of here: http://www.iana.org/procedures/nameserver-requirements.html

1
0
Silver badge
Thumb Down

Ugh, "Cybergeddon."

Just the link makes me cringe in disgust: "The great DDoS Cybergeddon of 2013." We're still here, aren't we? No one was hurt, right? Maybe spam increased for a few nanoseconds while Spamhaus was unreachable for a short time? Do I need to warm up my snow blower to deal with the deluge of junk e-mail?

Seeing as I'm posting this to a Europe-connected network that was supposedly strongly impacted by this, I'd say this was a storm in a teapot. But SANS has a history of sensationalizing internet events. They want to be the weather.com of cyberspace and it's embarrassing.

1
0

Re: Ugh, "Cybergeddon."

Did you actually read the article? They make it very clear early on that:

"Here is what did not happen: the Internet did not come close to coming down, not much real impact was felt outside the victims and those in close Internet-proximity to them and we were all still able to get to pinterest and see cat pictures online."

I got the distinct impression that the title was meant to be ironic, the piece itself was (like Trevor's, I should say) both informative and interesting, especially to someone who doesn't know the subject well (ie, me!)

3
0
Gold badge
Pint

@GregC

If my blitherings are interesting then I fear a walkabout outside where the daystar is might be advisable. It seems you need some of those photons that the great big ball of fusion in the sky spits out to help you create some vitamin D and jumpstart the "removing crazy" subsystems. That or oh look it's beer o'clock on a long weekend, bye!

1
0

Re: @GregC

No need to worry on my account, all things in moderation! I'm now geeked out for the week and will indeed be enjoying the sunshine at the weekend.

Nope, hang on. I suspect I'll be frozen solid by the time Sunday's done...

2
0
Gold badge
Pint

Re: @GregC

So long as you have a good weekend, sir, then all is good. Cheers and beers!

0
0
Thumb Down

Then SANS needs to stop cheapening Armageddon. And Pearl Harbor. And September 11th.

Catch phrases that invoke disgust:

"Digital Pearl Harbor."

"Digital Armageddon."

"Digital 9/11."

No one died due to any event reported using these catch phrases. Real people died in the real events. That is my problem: Cheapening the memory of real life events that killed real people.

If they want to stop disgusting me, SANS needs to stop using them.

1
5
Silver badge
Big Brother

I shall have to demoralize you...

Unless you are a born-again X-tian, Armageddon is not a "real event".

You may also notice that "Pearl Harbor" and "9/11" were attacks on Amurrica used at full efficiency to rally the rubes behind fascistic agendas of full-scale domination, so in a sense "digitalizing" them is fully in-line with we are looking at here.

You are also missing "Digital Attack on the Maine" and "Digital Gulf of Tonkin".

2
3
Gold badge
Pint

Re: I shall have to demoralize you...

Far more tragic would be the digital death of our beloved Playmonaut; a tragedy of intertubes proportions which would cause Register readers to rise up in droves against the evil aggressor.

R.I.P. Playmonaut.

0
0
Thumb Up

A++ read

It's worth 50 bits of click-bait about Stephen Fry or Freetards.

13
0

Wait...

You put all your traffic to your 'datacenter' through *one* server? I bet you run SBS as well.

1
1
Gold badge

Re: Wait...

There are redundancies. It's actually a cluster of 2 devices. I didn't really want to get that deep into it though. I wanted to talk about the DNS not setting up a cluster in CentOS.

1
0
Boffin

never "forget" any edge system!

Since my first time as victim of a hack was because of out-to-date BIND somewhere later in the nineties when I was really just a junior apprentice admin, I'd never "forget" what was running at the edge somewhere. It was Christmas morning though and I had to try to investigate from home using shaky ISDN dialup, no physical transport being available and trying to remember various IP addresses since all DNS was down. But the lesson I learned there was that the first machines I'd upgrade or keep at least to a recent level would be the ones opened up to the Interwebs like that. That's the only flaw I could find in the article's analysis, to have the priorities really ass backwards as it was stated that almost everything was gone over to CentOS 6 apart from one of the main attack vectors in the network! Not knowing or investigating BIND's defaults seems nearly irrelevant (although might have helped). Better have a proper upgrade policy and inventory in place. That's where proper management and security starts but definitely will not end. But the scrubber might have seen like a hassle to upgrade?

2
0
Gold badge

Re: never "forget" any edge system!

RHEL 5 is still under active support. There's nothing wrong with using it in live production. The system is updated religiously. The hardware refresh would have taken it to 6, but that got knocked back by about 6 months into April.

So while I may have forgotten which OS the little blighter was running (my mental filing system had ticked it over to "CentOS 6" already) it was still running a maintained, patched, and secure OS.

0
0
Linux

Re: never "forget" any edge system!

RHEL (and CentOS) are meant to be used in enterprise environments where things don't change quickly. This means Red Hat maintains an "old" version of a software component way after others have upgraded, religiously backporting patches to make sure it's secure. In other words stuff like BIND and openssh will have ancient version numbers, but still be up to date.

If you want cutting edge, either compile the software yourself or use another distro. I like CentOS myself.

0
0
Go

Here's a workaround on Windows Server

This was unexpected; Thanks for coming forward. It made me brave enough to come forward with a similar workaround I did on my Windows Server setup.

I publish a small handful of domains on that HP Microserver I bragged about a few weeks ago. One of those domains is an Active Directory domain as well as a publicly visible domain, and they handle internet e-mail and other internet things, so the domain controllers and dependent servers need to do recursive queries on the DCs running DNS, as well as host the DNS zones that make AD possible.

Windows doesn't have an IP access list saying who can do recursive queries and who cannot. But nothing stops you from copying the zones to another non-DC DNS server and disabling recursion on that.

I already have a reverse proxy server for various things; I just added DNS to it and port-forwarded DNS connections to it instead of to one of the DCs. Then I set up secondary copies of all my zones on it, and disabled recursion. I now have an edge DNS server that doesn't allow recursive queries and still acts authoritatively for my zones. I can still permit zone transfers from it to authorized servers outside as well, and do notifications of zone changes.

Sure, this is, 'duh, captain obvious' stuff for some. Who would have thought DNS would be used as a DDoS vector though? If Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott can come forward with this, so can I.

1
0
Gold badge

Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

Surely I'm not that negative! I like stuff that actually works just fine. I get tetchy when it doesn't do what it is supposed to. Or costs more than having a human do the same job. Or I haven't had coffee...

9
0

Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

That was the impression I got here and here. Who in these forums brought up the impact of first impressions?

0
0
Gold badge

Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

I calls things like I sees 'em. Good or bad. I don't hate any technology - except the things that let you robo call people in the middle of the night - but I do hate it when technology is badly implemented. A great example is saying "Trevor hates Windows.' I don't. Not even a little.

Oh, I hate lots of things about how Windows 8 and Server 2012 have been handled, but this doesn't mean I hate all of those operating systems. Nor does it mean I think they can never be made to not suck in future releases. I fact I have litterally begged Microsoft to make the relevant changes...because I think Windows is a useful tool.

I hate Microsoft's licensing department. I love Microsoft's storage team. I hate some of the very strict rules that Spiceworks has surrounding community interaction, but I love the opportunity it presents me to interact with other sysadmins and vendors.

You really, really, have to do an awful lot to get on my personal hate list. Even Oracle isn't on my "no buy, ever" list. And they take hostages! Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.

Technology is a tool. Corporations are groups of people each with individual hopes, dreams, goals and ideals. If technology sucks then I'll pan it. If it's great, I'll praise it. If it's boring, I probably won't even write about it. I'm harsh. I'm honest. I'm as up front as I know how to be. That's my job after all...

...biting the hand that feeds IT.

13
0

Re: Here's a workaround on Windows Server

http://support.microsoft.com/kb/198408 <<< theres some hidden gems within!

0
0
Anonymous Coward

Re: Here's a workaround on Windows Server

If you just "need to do recursive queries" for "internet e-mail and other internet things", then you should consider removing all the root servers from your root hint list, and just using your (local and) upstream provider(s).

This doesn't make the problem go away -- you can still get rogue external requests -- put it translates them into a problem for your upstream supplier. In a DDOS situation, (1) your upstream supplier will have (hopefully) respond from cache, and (hopefully) have better quallified 24 hr support, who can pull the plug.

0
0
Bronze badge

Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott

Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.

You can add me to the club of people who have a severe disdain for anything SONY for exactly the same reason (a rootkit infection that killed a system).

0
0

filter on sane sender address?

Do ISP filter *outbound* traffic to ensure that the sender's address is sane?

The filtering could be done around the NAS level, or at the peer-exchange (LINX).

0
0
Silver badge
Boffin

Re: filter on sane sender address?

Most, no.

should they? yes.

1
0
Bronze badge

Kessel Run?

"it averaged 500Kbit/ses"

For how long? That's a speed, not a volume.

1
0

Page:

This topic is closed for new posts.