US broadband providers holding last-mile traffic ransom, ISP alleges Consumer broadband providers in the US are deliberately ignoring traffic congestion on their networks, resulting in dramatically degraded service to customers, one major ISP has claimed. The issue, according to a blog post by Level 3 Communications VP of content and media Mark Taylor, lies with the peering interconnects that …
Those greedy bastards are raising their rates while providing as little service as possible (and paying their top execs big bonuses). Many of them have local monopolies in the areas that they service, so they have no incentive to provide improvements ("Don't like our service? Then go with another cable TV company for your Internet. What's that you say: there isn't another cable TV company at your location? Gee, that's too bad. Hahahahaha!!!!").
</rant>
"Many of them have local monopolies"
Are there any in the US that *aren't* unless you buy Dish? I was even surprised that despite the Google hooplah about how gigabit will help competition, that Austin is just as carved up into sub-fiefdoms as anywhere else. This block, you get Google Fiber. That block, you get AT&T gigabit. Next over is the local Austin gigabit provider. The other block, yer fucked, bro.
Shoot, here at Brighthouse/TWC in Orlando, I'd be happy with consistent megabit service.
There are a few places where the local city government has installed their own metro-area network that sells TV, Internet, and phone services to its constituents. The cable TV companies are very upset about having direct competition at those locations. But this is by far the exception, not the rule. :-(
Yeah, that makes sense. But still it seems a bit like Level 3 is trying to overstep their bounds by (almost) naming-and-shaming. I mean, if they were in a court case or something, and this came out as part of their testimony, I could understand. But to just volunteer the info out of the blue as a way to nudge the ISPs into buying fatter pipes, eh, that seems a bit much.
Level-3 just wants larger transit pipes so they can sign more customers up. Seems to me that Level-3 wants the last-mile ISP's to upgrade their network to support Level-3.
Is ISP Z has a peering link with Level-3 that is 10Gbps and it is running at 90%, that is 90% going towards ISP Z. What needs to be known though is what is the utilization on that link going towards Level-3. What if it is 10%? ISP Z is receiving far more traffic than what they are giving to Level-3 and that peering interconnect seems to favor Level-3 far more than ISP Z.
Level-3 is known to offer service at a far lower cost. They seem to look for the more profitable side of the business than the last mile portion. I just don't think Level-3 is telling the whole picture.
I also wonder how many complaints are from residential customers that have purchased an Internet package that is not what they really need. Netflix can easily take 5 to 7Mbps and if someone gets a 6 or 10Mbps package, that is just not going to work if they are trying to access the Internet while they watch TV.
And that's not a coal dig at Level 3, but it's nice they're getting back to their routes.
So LVLT's dude says they've got 180,000 miles of fibre. Whoopee. How much of this is lit? This is an old ISP trick. Say I want to do East Coast US and run ducts from NYIX to Miami's Nap of the Americas. That's 1,300 miles. I run a 288f cable down part of 1 duct and my network's got 365,000 miles of fibre! So a fairly meaningless metric when a 288f cable's only about 3/4" in diameter.
Then it's about money. Level 3's "original investment capital" was $40bn. But Level 3's actual investment was much smaller given it's had a habit of buying up distressed assets at a fraction of of the original investment. This includes stuff like their Transatlantic cable, which is old and limited capacity. By way of contrast, Verizon has reportedly invested $23bn in it's FiOS network.
Then it talks about it's thousands of customers, and only 51 peers. It likes customers, they pay. It doesn't like peers as much. It's de-peered another familiar name in this dispute, Cogent in the past over asymmetric traffic ratios. Then changed it's peering agreements from ratio-based to a bizarre 'bit-mile' model that naturally suits it's own network.
So people talk about double-dipping, not realising this happens on the transit side, and can be a good thing. If you're an ISP who's one of the 51 peers, you may not pay transit charges to LVLT. Netflix pays LVLT for transit. LVLT charges a non-peer ISP transit. Transit rates are usually priced per Mbps and based on committment or volume. Netflix probably pays a much lower rate per Mbps than LVLT charges a small broadband provider.
The small/regional broadband providers don't get much choice other than to pay some ISP for transit. LVLT links to their standard peering agreement-
"must have a backbone network node in at least six of the nine U.S. Census Bureau Divisions in the United States (New England, Middle Atlantic, South Atlantic, East South Central, East North Central, West North Central, West South Central, Mountain, Pacific);"
So it would be rather expensive for the 1 European congested network to meet LVLT's US peering criteria, especially when LVLT expects reciprocal settlement free peering in their own market, regardless of cost. This has always been a challenge for non-US ISPs when historically the majority of the popular content was hosted in the US. But many eyeball networks have little choice but to pay someone for transit (like LVLT) or build out their network using wavelengths (from LVLT) and then spend time/money peering.
But this article was written by LVLT's Content VP, who points out they have thousands of Ethernet connections. Some of which would be for other content providers, businesses, VoIP operators who may be wondering why there's so much congestion on LVLT's network. Those customers may not care about streaming video but wondering why their customers are complaining and be looking to switch ISPs. This is where disputes between the likes of LVLT and Verizon or AT&T get more nuanced as they compete across the portfolio. With regional eyeball networks, LVLT (and Cogents) whines smack of self-interest.
As Netflix builds out it's network, it's probably starting to realise it can get better customer satisfaction by cutting out the middlemen by getting closer to the eyeball networks and paid (or even settlement-free) peering. Providing it costs Netflix less than they pay their transit ISP(s), they're ahead, and if it costs the eyeball network less than they're paying their transit provider, they're ahead as well. And both Netflix and the eyeball network's customers get a better experience. Naturally LVLT doesn't like this future-
Removing these middlemen would leave a massive hole in the Internet.
...P&L on LVLT's balance sheet. But as anyone with a sense of Internet history knows, it routes around problems. There are also some other fun challenges, like the 100Gbps Washington connection used as an example. LAG on 100Gbps links gets interesting, and IP balances flows about as well as a drunken sumo wrestler on a pogo stick.
Level 3 wouldn't be paid more for the bigger pipe with the ISP (usually) as it would (typically) be a 'settlement free peering'. However, at the other side of their network they have content providers who are paying and they don't want them pissed off that their traffic is being throttled on peering points. And bigger peering points means more traffic from the ports on their network where they do charge...
It'll probably be framed to reflect on a more general fundamental principal of the great nation of the US!
"What we're doing is providing services in accord with free market principles and the capitalist ideals that made our nation the best in the world! Despite recent attempts to paint us as some kind of villains in this matter, we feel confident that our customers will see that we're upholding grand American traditions."
Or something to that effect...
Perhaps they will use exactly the same argument L3 used when they depeered cogent for exactly the same reason the cableco's aren't increased the ratio caps with L3? Whilst I don't doubt for a second the cablecos are doing this for malicious reasons, they are doing it behind a cloak of reason. The very reason L3 used to defend its own business model from attacks by bottom feeders like Cogent. Peering is supposed to be mutually beneficial. Transit is there for when it isn't. If L3 sold transit to Netflix for dirt cheap rates, undercutting what they would have paid other Tier 1 ISP's then why should Cox et al offload that additional traffic for free?
Because the "eyeballs" are paying $50-$100 a month to their cable provider to deliver that data to their homes. The "broadband providers" will always be "consumers" at the transit points, and they shouldn't be allowed to charge both the senders (Netflix, etc) and the requesters (Joe & Jane Sixpack) for the same packets!
Frm the mrtg graphs posted it showed a traffic ratio of 5:1. Whilst virtually everything said about the cableco's being gimps is true, there is an underlying truth to their position on peering vs transit. Peering is supposed to be mutually beneficial, otherwise you buy transit. There is normally a ratio set at which point the agreement can be dropped. It can be pretty generous, 3:1 is about the maximum I have seen for normal agreements. I can kind of see the argument from the cableco, if another ISP undercuts you on price then expects to dump the traffic on your network for free that is a little tight. If you cannot do your fare share in the peering agreement, be in acting as a long haul jump or last mile, then why should an ISP continue to act as your last mile. Once the benefit to the other party declines you need to buy transit rather than peer.
The 'upgrades' people keep mentioning are not to backbones, routers etc, they are simply more ports or higher port speeds at free peering points. They could get this pretty much for free at public exchanges but it would be likely to offer poor performance as the partners who would want to peer would probably not be last mile ISP's.
The reason that larger ratio's are allowed (than 1:1) is because it can be beneficial to one ISP to have access to certain routes to improve performance even if it means accepting an unequal amount of traffic in return.
Cogent ran into similar problems, massively undercut established ISP's (literally by an order of magnitude), dumped the traffic onto their relatively impressive backbone then find you actually have to be able to get rid of that traffic. They had peering arrangements but frequently overran their ratios and got all upset that they got depeered. <irony> L3 was one of the companies that depeered Cogent for exactly this reason (Sprint, Telia and a few others also depeered cogent). If the agreement is not fair then you need to buy transit, that is the way the market works. It might not be fair in some peoples minds, it might need changing, but you cannot do exactly what you complained about another company for doing then cry hard done by. You either agree that peering should be unlimited and give up on selling transit, or you live by fair ratio rules. If you are going to ride share the school run and all of a sudden find yourself doing 4 days a week and the other family only 1? You might have your hand out for gas money no?
All very true: Netflix and the like provide the majority of eyeball bound traffic in the US. Why shouldn't they pay carriers for the massive amount of traffic that they are dumping on them?
If the US wants to improve the situation, de-regulating the ISP business in the US is a better solution: make it easier to lay fiber in the street and gain operator licenses.
The current US peering practice of emphasizing private peering links rather than large public fabrics can hardly help either :-(
"All very true: Netflix and the like provide the majority of eyeball bound traffic in the US. Why shouldn't they pay carriers for the massive amount of traffic that they are dumping on them?"
should my ISP be paying me for the massive amounts of data I request/they dump on me, or would that be silly?
"Why shouldn't [Netflix] pay carriers for the massive amount of traffic that they are dumping on them?"
For the same reason that if i pay for a 100Mbit service I should not be subject to throttling or additional charges if I decide to use 100Mbit 24/7. Netflix will be paying to have a connection to their CDN, if the people selling that connection failed to price it correctly or are worried their network can't cope without more investment then it is their problem, not Netflixs'.
Nobody is artificially throttling Netflix, at least not directly. The intermediate carrier simply does not have enough network egress capacity to the last mile isp's. They can pay for it or Netflix can directly. Now the last mile carrier could be deliberately not sending add much traffic as they could back over the peering to artificially inflate the ratio, I certainly wouldn't put it past them, but I suspect it's simply a case of them not having the volume given the natural disparity with VOD.
The bottom line is the last mile isp's will make X amount of money, no matter where you cut their income they will make it up elsewhere. Either Netflix pays which means you pay, or an intermediate carrier pays and passes the charge onto Netflix who passes it on to you or the last mile isp just ups your monthly fee and you pay directly. Just like banks, they will always find a way, no matter what you stop them charging it always ends up costing you the same overall.
As regards paying for 100mbps. You don't, you do not have a leased line, you have a contested connection that at best should work at that speed within their network. You are not guaranteed a link of that speed to any other server although you may get it. The cost had always been borne by both ends of the connection based upon relative volumes. (Baring Oz which at least for a while only charged for one way iirc).
Is just how it is, it could change but it won't make your bill any cheaper.
Bottom line. Level 3 under bid and cannot get rid of the data they have so are having to consider paying for transit because a last mile provider won't continue to take it for free in ever increasing volumes. Cogent did exactly the same and l3 complained about it, but now is ok, I guess they can both look down on HE now.
I like Netflix, it uses a lot of data though and if it costs me an extra dollar or two a month (directly or indirectly) it isn't the end of the world.
Netflix already pay their share. As do all content providers. What you're basically saying is that everyone with a website should pay every ISP to make sure their site is available to that ISPs customers. You already do that, by paying your hosting provider to carry your traffic, they in turn then peer with other providers and buy connectivity from other providers such as Level(3), Cogent, NTT etc.
Not what I am saying at all. As it stands Netflix have a contract with level 3 so the immediate solution is level 3 need to pay for transit to the cable cos et al, or try and negotiate more peering which is highly unlikely to happen.
The discussion is regarding this being a net neutrality issue, which or is not. It is simply a capacity issue, Netflix could be sending any kind of data, if the volume was the same the result would be the same. Peering only works where it is mutually beneficial, you cannot build a business plan based on undercutting tier 1 isp's then trying to dump it for free on the same isp's you just undercut. Remember Netflix is paying level 3 and no doubt others to perform a service, exactly the same add cogent 's customers did, but in just the same way the do not have the capacity to get it to the required destination and have maxed out their free peering agreements.
When you run a server or servers you pay (usually via a DC) one or more isp's to get your data to the end users, you don't need to pay every isp, but the isp's you do pay have an obligation to actually supply the routes they sell. If they have congestion they need to add capacity or route out another way. Level 3 is saying that the last mile networks should accept higher peering ratios for no other reason than it needs it to keep selling huge amounts of transit for cheap.
If the mrtgs showed parity or even closer to it they might have an argument but if they showed parity I doubt there would be an issue. This isn't net neutrality, this is basic economics, they sold something they couldn't deliver and aren't prepared to pay to make good on their deal.
That is one way of looking at it (and no I haven't forgotten that at all and addressed it above), the other is that the money paid for data ingress subsidies the cost of the home broadband connection. The same model is used by every single provider of any size. Smaller ISP's such as individual towns with their own ISP may mix free public peering with paid transit because they don't have the scale to justify peering with major telcos but besides that every last mile ISP does the same. Last mile providers not only have to manage the backbone that other providers do but the last mile network, hence larger charges.
You also neglect to consider that not all that data will go to that telcos last mile customers, plenty of it will be going to another ASN either within the country or abroad. Parity is not distracting when it comes to peering, it is essential to how peering actually works. Whilst there can, and usually is, a level of disparity it cannot be too extreme otherwise they would never sell any transit and they would recover the cost from their residential subscribers. This balances the cost based on use.
It is also important to realize that the ISP's are not targeting Netflix, they are not filtering or blocking ports etc, this is nothing more than Netflix using an intermediate ISP who has maxed out their interconnects (which affects all traffic equally) and is refusing to pay more and trying to elicit sympathy from the public by hoping they won't realize the con.
Forcing last mile providers to accept all data ingress gratis could happen, or the rates could be fixed (although not in the USA, no chance of that happening) but I can't believe anyone would believe this would result in anything getting cheaper? It would just shift the cost elsewhere, anyone who thinks that the ISP's would just shrug and make less money is momentously naive. If anything they would use it as a way of making even more money, hiding an increase in new charges. The current system actually works, both ends of the chain pay an amount based upon roughly the amount they use sharing the cost of the end to end infrastructure plus profit to the providers.
If the ISP's were actually deliberately blocking on QOS'ing Netflix then that would be unacceptable, unfortunately Netflix just uses a hell of a lot of data so is being hit hardest by the reality that no matter how you try and avoid it using intermediaries, you are going to have to pay the piper. You can negotiate to peer directly or ensure intermediaries have enough peering or transit capacity. In the end it seems they are paying to peer directly.
by legislating the whole question including rates.
Now our cities are being dug up as competing fibre optic vendors lay cable. Of course, this excludes Rogers Cable, they prefer the visual pollution system where all there big, black, cables are strung from utility poles.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
