Can ISPs step up and solve the DDoS problem? Solve the DDoS problem? No problem. We’ll just get ISPs to rewrite the internet. In this interview Ian Levy, technical director of GCHQ’s National Cyber Security Centre, says it’s up to ISPs to rewrite internet standards and stamp out DDoS attacks coming from the UK. In particular, they should change the Border Gateway Protocol …
Well, what made the Dyn attack so difficult to stop was that it was almost perfectly disguised. It looked like a completely-legitimate HTTP request for an actual existing resource that's not a malicious payload in itself. That's close to passing an Internet Turing Test in that regard, meaning a future attack will probably be even harder to filter because next time they probably won't use just one resource but a bunch of common ones. Also note that Dyn had fallback systems that themselves got swamped. An attack that can defeat redundancies, and yet it was just a dry run.
I'm actually curious, is there a reason that ISP's havent already implemented These "easy" solutions mentioned in the article? I know the obvious answer is going to be laziness, but lets be honest thats rarely the case. Usually it comes down to one of either:
- too expensive,
- too difficult to implement,
- too specialist for your Standard IT guy to implement,
- too time consuming to implement,
- conflicts with other Equipment,
- or a belief that it wont have any actual effect.
So what is the reason why ISP's havent implemented These Solutions on mass, so far? There HAS too be a reason...
I suppose an ISP's business model is selling connectivity to punters at a lower price than the next ISP. Service enhancements that don't directly improve the offer to consumers are just a cost. It seems to be a very competitive business, so there probably isn't much margin available for this kind of thing.
Anon because of where I sit.
The general reason is "keeps getting pushed down the technical/financial/project management priority stack".
"Show me the quantifiable benefit to my business of fixing network vulnerability X, compared to the already-quantified benefit of launching wizzy new service Y."
Even if X does have a quantifiable benefit, it will still go into a looong queue.
ISPs are the same as all businesses: constantly juggling how to allocate resources (and of course, internal politics is a large part of that juggling)
Probably simply because it costs money, there's not immediate payback for the expense of fixing what they see as an outside problem. Realistically too, they could spend the time and money to "fix" these issues and then, most likely, the next attack would work around it.
However - since the government is mandating the ISP's record virtually every communication from their users, the ISP could have quite a lot of data that would help them figure out the sources of the problems. Back in the old days, if you were running a piece of kit that interfered with your neighbours TV reception, someone would come around and have a word with you.
Back in the old days, if you were running a piece of kit that interfered with your neighbours TV reception, someone would come around and have a word with you.
Ah yes.. I used to work in a related field in the early 90's. Someone had a cordless phone that was, well, "not compliant with national standards". This was located between the city's main transmitter and a local translator, which also on-fed a few other translators, so a large area was getting this pricks interference.
We were discussing it with BCL (NZ's version of Aunty back then, or close to - managed the infrastructure, licensing (yes we had that silliness1 at one time!) etc), and they supplied us with a pile of tapes and a couple of standard VCR's and asked us to record the interference any time we saw it. They claimed they had equipment which would let them tune in to the phone calls, and they'd be able to listen until the person gave out their address or name. Seems it worked. The interference was green and red or purple horizontal bands on the picture, maybe 10 or so (5 each, alternating) which would move up and down slightly. You could kind of see a speech pattern in the rate of movement of these lines (imagine something like the lines on the edges of the screen when a Speccy was loading a program from tape)
1 I wasn't sure on the spelling, though my spell check says that's right. I pasted it into DDG to check. First result for "silliness"? An add for facebork sign-up!
It's to do with externalities, an economic problem with an economic solution. For an individual ISP, implementing BCP38 on your network helps other people, but costs you and doesn't help you directly very much, unless everyone implements it. It's a bit like treating a river or the air as a free place to dispose of waste which you're not very interested in. If a Victorian mill owner killed a fishery downstream that was someone else's problem. When his grandson cleaned up the mill effluent in response to legislation, everyone benefited when the river came back to life again. Clearly it would be preferable if the ISP industry could self police this problem - compared to politicians having to deal with this problem by legislation and international treaty as the latter mechanism progresses much too slowly if at all.
The mechanism for ISP community self policing concerns the conditions ISP networks have to maintain in order to be able to obtain the mutual respect of peers or access peering exchanges such as LINX on cheapest possible terms or at all, based on membership agreements enforced by contract law. This probably requires peering exchanges to agree common standards between exchanges in different countries and with backbone providers in a similar way, through club memberships where membership comes with agreed benefits.
If the industry fails to act in this kind of way, the public interest will eventually require the legislation and treaty route to ensure that the industry does act to ensure the polluter pays, as has occurred and is occurring with other pollution externalities.
There may, for some ISPs, be a strong incentive not to implement BCP38 filtering: they have paying customers who want to send packets with forged source addresses. So the ISP is not generating malicious traffic, but is paid to condone it by doing nothing. Being paid to do nothing is a powerful incentive.
There's also a technical excuse: if you have multi-homed customers, BCP38 might block legitimate traffic. RFC8028 is supposed to help with that for the IPv6 case, but isn't the whole story, and it's hard to fix for IPv4.
Basically, for "broadband" type connections, it's perfectly fine. I'd expect any sane ISP to do it, and many if not most do already.
For anything more fancy, a lot of stuff there are perfectly good reasons for wanting to do starts breaking and potentially in very "interesting" ways (partial reachability, yummy troubleshooting!).
And as soon as it starts being called "transit" and not "internet connection", you can essentially forget about it. You basically have to do this at the very edges of the network - and that might very well be deep inside a customer network, far out of ISP control.
Like the previous poster said, IPv6 is supposed to let you multihome without having to essentially become your own ISP and has some stuff that could help with this, but I'm not holding my breath...
And let's not forget - none of this actually solves the problem of DDoS or would even make a major dent in it even with 100% adoption.
For the n:th time, this is not a solution to DDoS. Has anyone even bothered to check whether these recent large high-profile attacks were spoofed at all?
BCP38 was originally intended to help stop simple SYN floods - which was solved with SYN cookies and similar technologies instead. Spoofing or not does not actually matter for DDoS attacks as long as your botnet is sufficiently large, and in fact, not spoofing lets you do a lot more fancy and effective stuff like application-level attacks.
The only attack that actually requires spoofing is amplifier/reflection attacks, and all that's needed for those is a single, or a few, hosts that can spoof which will always have to exist.
Many if not most ISPs of the variety "Selling end-user access" have already done this. It's part of the standard configuration of much access gear. However, at the backbone level - or even for relatively common things like many multi-homed business connections - this can not be done. There is no guarantee, or even intention, of routing always being symmetric. This is fundamental in how internet routing actually works. Each hop independently decides where to send traffic to a specific destination, without any regard for what decision the destination in turn would make to reach the source or even whether it agrees the address is reachable via that path at all.
Unless you are suggesting that we replace IP with something different (X.25 perhaps?), then it CAN NOT be done universally. AND it's largely already being done for the case where it can, and obviously that hasn't solved the problem or even made any meaningful difference.
What perhaps needs to be done is improving the ability to easily get ACLs added at various points in the global infrastructure to help mitigation of really large attacks, but this is a very hard problem to solve both politically and technically (probably still a lot of carrier gear that can't do meaningful ACLs at wirespeed, and unless there's a surefire way to authenticate these requests across the global network it would in itself add a very effective DoS vector).
To continue my previous question, since the vast majority of answers from you knowledgable Folks is Cost - What costs are actually involved? I can think of the labour costs of getting the engineers to implement the various strategies, but what else? I dont work in IT, but it sounded from the article that it would not require updating kit or anything like that to implement these strategies, so I was just wondering what the costs would be?
And wouldnt this be offset by the cost of reduced data being passed along to your Backbone Providers? Or am I talking bollocks there? ;)
See my previous post - the cost would be _your customers perfectly legitimate traffic getting dropped on the floor_ if this was implemented in the places it currently isn't.
But to answer your second question - it's unlikely that DDoS attacks originating from your customers would noticeable raise bandwidth bills. Even if you have a lot of infected customers with pretty hefty pipes, chances are none of the individual hosts will be maxing out their pipes for very long - people tend to notice "the internet not working/being very slow". And if the aggregated attack traffic is enough to significantly affect transit pipes action would be taken rapidly by any half-competent ISP.
Being on the receiving end can be costly though, but if you are running some sort of service you often have a lot more outgoing than incoming traffic so it takes quite a bit for an incoming DDoS attack to start affecting your bill.
The cost is pretty small when you get down to the bottom layers. Let's say my ISP wants to fix this. I have a DSL connection and have an IP address x.y.z.n where n>=2 and I get a gateway address x.y.z.1. If that gateway refuses any packets from me and anyone else connected to the same gateway with a source address that isn't x.y.z/24, it should be dropped. Sure, they probably have a million DSL customers, and thus thousands of gateways. Surely they have an automated system for deploying changes - if they wanted to make some change like blocking port 25 they aren't having someone login to each one to make changes by hand. It should be pretty simple for them to deploy this filter across all their customers.
If my ISP doesn't do that, once my packets leave my ISP it would be WAY harder for the upstream peers to determine if I have sent packets with forged source addresses, because of how disjoint the IP space is. It isn't like they can filter my ISPs packets with a simple single rule, because they aren't going to have everything in a nice neat little netblock. And that doesn't even get into ISPs serving customers who have their own class C (I actually have my own, assigned by ARIN in the 90s, but I only use it internally)
What those higher levels can do is put progressively stiffer penalties on downstream peers that let forged traffic through. When an attack is identified as coming from me, my ISP would be penalized for not filtering it. Start small but make them go up by time/frequency, and soon the excuse "it costs too much to do" will be replaced by "it costs too much NOT to do".
The problem is, you need these penalties everywhere - it doesn't do much good to fix it in 90% of the world, it has to be everywhere. So it isn't a law in the US, law in the UK, etc. It has to be something enforced by the IETF/IANA/etc. type body - threaten to cut off their DNS delegation if they don't comply.
Chances are more than decent your ISP already does this - in fact even that they do it to the point where traffic from any IP address(es) not actually assigned to you is dropped, not just from a subnet other than the one you reside in.
While it's not 100% (though probably fairly close for anything resembling major consumer providers in Europe/US), it's certainly already done to the point where no attacker with a botnet consisting of residential customers would actually have them send spoofed traffic.
(And just a minor nitpick and/or possibly slightly relevant detail, subnets and gateways tend to have little to do with the physical network layout when it comes to this type of access networks, by the way...)
Mr. Picky would like to point out that X.25 is/was usually only between the end client and the exchange.
Inside the network is usually X.75 which (IIRC) does much the same as TCP/IP in scattering individual data packets all over the network then re-assembling them at the other end.
Which seems to tie in with your description of TCP/IP routing not being symmetric.
It was just a modest proposal :-).
I've only used it as an end user, but if I have understood things correctly, once a virtual circuit is established in X.25/75 the packets always follow the same path.
Plus routing is probably a lot simpler (or rather dumber) than BGP? As in, more based on static routes with configured failover than having the network figuring out paths on the fly. Seeing as it's coming from the telco world and all...
PS. All-Mighty Wikipedia claims that the only difference between X.25 and X.75 is that network status messages (clear/reset) can be sent by both sides of X.75 while in X.25 only the network side and not the subscriber can do it.
Those of us who actually tried to make X.25/X.75 work in the 1980s are still very glad that it went away. IP had the advantage of actually working when you plugged it together, and the related feature that anybody could send anything to anybody. It's that feature that allowed the invention of the Web, and that allows DDoS today. It comes with the territory. (The old PTT monopolies knew this very well - it was their main reason for pushing the virtual circuit model, in order to preserve their monopolies and their revenue streams. X.75 was invented precisely to interconnect national monopolies across frontiers.)
We aren't even anywhere near ISPs working to make the situation better, they can't won't even stop making thinks worse. Look at Talk Talk, they've been told 76,000 of their customers Wifi passwords are circulating on the internet, and they won't even tell their customers to change them.
Those are some good ideas for fixes but all rely on herding the internet to do something en-mass - we've tried that with IPv6 and look how that's working out - yea, we'll get there one day, sometime, perhaps.
Probably the simplest fix would be replace all the users modems/routers with devices that the ISP could disable/rate-limit if excess/abnormal traffic was suddenly flowing. This model works with credit cards - you can buy a packet of fags every day on the credit card but try buying a 747 and alarm bells ring because that's not normal for you.
Nothing we do will be 100% effective, everything we do will inconvenience someone. We need to be realistic about this.
Which does nothing for a DDOS because every single endpoint is going under the radar. It's only at the final endpoint where everything comes crashing down. Just think Zerg Rushes or Human Wave tactics. One unit isn't much, but flood a place with a million of them all at once, all coming from different locations? Even better, think of a grand final game where people from all over congregate. Eventually, you put physics against the target as you're basically trying to cram millions of eggs into a package only built for thousands.
The ISP has no control over the end-point, the target of the attack - but it does have control over the gateway between itself and the Internet. So it it sees 10,000 of its customers suddenly start pinging one range of addresses then it would suggest that they have been compromised.
It can do something about that.
I'm not suggesting that the ISP can stop the attack - but it can steal the shoelaces from the attackers. It doesn't stop the attack but it's within the ISP's control to detect and not participate - at least to some degree.
The ISP has no control over the end-point, the target of the attack - but it does have control over the gateway between itself and the Internet. So it it sees 10,000 of its customers suddenly start pinging one range of addresses then it would suggest that they have been compromised.
AIUI, many of these DDOS-type attacks work on the basis of requesting legitimate traffic. Say smallbusiness.co.uk pisses someone off and that someone can bring a DDOS to bear on them. Instead of seeing 20 hits a day, smallbusiness.co.uk suddenly starts seeing 20,000,000 new visitors each hour, and as it's servers aren't set up to cope with that traffic..
OTOH, with the likes of social media, it's possible some website could have a page that gets mentioned on farcebook or twatter etc, seen by 70k subscribers of a large ISP, 10k of whom decide to visit the site for a looksee.
Something I would not want to be at an ISP in charge of figuring out. On the one hand, 10k visitors could swamp a small business and knock them off the web for a while. OTOH, they may've successfully marketed something and be geared up for such traffic. Without knowing the full chain of events for at least some of the ISP's customers, it would be quite hard to tell if it was a bot or a twat that caused the jump in traffic.
"The ISP has no control over the end-point, the target of the attack - but it does have control over the gateway between itself and the Internet. So it it sees 10,000 of its customers suddenly start pinging one range of addresses then it would suggest that they have been compromised."
And there's squat all the target ISP can do about it because the volume gets so massive the bottlenecks move up the chain. Sure, they know there's a problem, but it's already upon them and there's little a site can do versus a sheer legitimate traffic flood other than to drop everything on the floor, which is just as bad for your business and trying to beat back the horde. It's like with a mudslide or avalanche: whether you're buried under it or carried away by it, your day's pretty much shot either way.
Part of it is simple - don't allow forged source addresses to escape. You probably can't fix that at a per user level, though it can't hurt. Everyone gets their internet from some sort of provider, it is those providers that directly attach to end customers that can EASILY fix forged addresses. Put economic pressure on them (fines or cutting them off from DNS delegation) to force them to comply.
Fixing a DoS that relies on massive amounts of data from a massive amount of sources is a whole other problem. I don't have a fix for that (start writing better software...yeah right!) but we can at least lick amplification attacks fairly easily and halve the scope of the problem.
Costs too much money. ISPs are like any other business: driven to raise profits, and these kinds of things are externalities that can be easily converted into Somebody Else's Problems. Heck, current business law (which is less likely to be changed than the flaming Inferno freezing over) demands deflection of responsibility because otherwise investors won't invest.
Anti-spoofing would be no help whatsoever with Mirai / the current Internet of Trash botnet issues. And DDoS are the loudest, most easily spotted and mitigated forms of attack and monetising them is becoming steadily harder and fewer as fewer victims pay up and the specialist DDoS defence orgs scale out and get better at it.
"Perhaps some government regulation is appropriate..."
If government regulation is the answer, then you're asking the wrong question.
Government regulation usually involves egregious abuse of said regulation further down the line. RIPA being used to spy on what people put into their waste bins for example.
"If government regulation is the answer, then you're asking the wrong question."
Well then, who else can run the jails? Certain necessities of civilization are, in a monetary sense, sinks. Meaning there's no incentive for private enterprise to do it. Yet something needs to be done to control the genuine criminal element out to pilfer from the common man. If the status quo is unacceptable, and there's no money angle for the private sector, guess who's left?
So basically, if you don't like government regulation, what happens when it's the ONLY option left? If you can't trust the government at this point, you can't trust anyone, and that means anarchy.
"We can also encourage IoT manufacturers to impose better security in IoT equipment."
Things will be manufactured and (not) supported in a manner consistent with their plummetting cost. Vendors will focus on competing to get new features to market, not long term matters like security.
This is a very real challenge: in a world of open Internet access, and relatively free trade, it is very hard to discourage the public from buying cheap electronic baubles sourced from vendors in far-off jurisdictions.
"We can also encourage IoT manufacturers to impose better security in IoT equipment."
The IoT maniufacturers will be driven by competitive pressures to get new features into the market first, not to worry about security or support.
This is a very real challenge: in a world of open Internet access and relatively free trade, it is very difficult to discourage the consumer public from purchasing cheap electronic baubles, sourced from, and (not) supported by vendors in far-off jurisdictions.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
