back to article Want to beat Verizon's slow Netflix? Get a VPN

Yet another person has stepped forward to claim that Verizon is deliberately throttling Netflix traffic, this time with video evidence. Colin Nederkoorn, CEO of software firm Customer.io, alleged in a personal blog post that his Verizon Fios internet connection in New York slows Netflix traffic to a trickle, but runs at full …

  1. cyke1

    Pretty clear verizon is being anti-competitive. Probably trying to force people in to use verizon on-demand streaming options instead of letting that money slip away to netflix.

    1. Anonymous Coward
      Anonymous Coward

      They are trying to force Netflix to buy pipes direct from them (Verizon). Verizon wants to make money from Netflix; that is what they are after.

    2. Anonymous Coward
      Anonymous Coward

      Nothing is being clear

      It would have been the case of "Verizon being anticompetitive" in Europe where most of the peering is public or through common exchanges. There the pipe to other ISPs is shared so if someone gets a preferential treatment it is clearly obvious.

      This is not the case in the USA. In USA most peering is private using dedicated links. In fact, some USA companies are running decade long strategies to deliberately sabotage any public peering development. The little public peering that there is still alive is around Seattle because of Google and MSFT sponsoring the exchange there.

      As a result you can (in fact you are likely to) have a congested direct link between let's say VZ and Netflix (direct or via Level3) while having a non-congested link between VZ and one of the Bells which in turn has a non-congested link to Netflix. Such traffic anomalies are natural for the USA Internet landscape. That is how it is being run today.

      Now why VZ is not increasing the bandwidth on the links between it and Netflix (via whatever route they happen to be) is a different matter. It may or may not be anticompetitive. This video is not a proof of that. It is a demo of how the USA internet works and why the lack of public peerings is a key factor in distorting neutrality and competitiveness issues.

    3. Anonymous Coward
      Anonymous Coward

      not so clear

      Actually, all this test means is that the default path between Netflix and this guy is congested, and that the paths between Netflix and the VPN host, and between the VPN host and this guy, are not congested. It does NOT mean that Verizon is the one congested, nor does it mean that Verizon is throttling traffic.

      Example: Path from Seattle to Chicago is clear, Path from Chicago to New York is congested, and the path from New York to Seattle is clear. If you are sitting in Chicago trying to get to New York, it can be faster to VPN into Seattle and route through there than going to New York directly.

      If you are trying to blame throttling without first finding out WHAT is being tested, all you are doing is waving a sign that says "the holder of this sign is uninformed and his opinions are FUD".

      (And for the record I neither use nor have a financial interest in Verizon *or* Netflix - I just have a background in Internet routing)

      1. Schultz

        Re: not so clear

        "Actually, all this test means is that the default path between Netflix and this guy is congested, and that the paths between Netflix and the VPN host, and between the VPN host and this guy, are not congested."

        And I thought the internet routes packages the fastest way, not the shortest way (for your personal definition of 'short'). Somebody somewhere is steering the traffic and is not doing a good / honest job.

        1. Anonymous Coward
          Anonymous Coward

          Re: not so clear

          "And I thought the internet routes packages the fastest way, not the shortest way (for your personal definition of 'short'). Somebody somewhere is steering the traffic and is not doing a good / honest job."

          You really don't understand BGP then. Here is how Cisco does it:

          How the Best Path Algorithm Works

          BGP assigns the first valid path as the current best path. BGP then compares the best path with the next path in the list, until BGP reaches the end of the list of valid paths. This list provides the rules that are used to determine the best path:

          Prefer the path with the highest WEIGHT.

          Note: WEIGHT is a Cisco-specific parameter. It is local to the router on which it is configured.

          Prefer the path with the highest LOCAL_PREF.

          Note: A path without LOCAL_PREF is considered to have had the value set with the bgp default local-preference command, or to have a value of 100 by default.

          Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP.

          Local paths that are sourced by the network or redistribute commands are preferred over local aggregates that are sourced by the aggregate-address command.

          Prefer the path with the shortest AS_PATH.

          Note: Be aware of these items:

          This step is skipped if you have configured the bgp bestpath as-path ignore command.

          An AS_SET counts as 1, no matter how many ASs are in the set.

          The AS_CONFED_SEQUENCE and AS_CONFED_SET are not included in the AS_PATH length.

          Prefer the path with the lowest origin type.

          Note: IGP is lower than Exterior Gateway Protocol (EGP), and EGP is lower than INCOMPLETE.

          Prefer the path with the lowest multi-exit discriminator (MED).

          Note: Be aware of these items:

          This comparison only occurs if the first (the neighboring) AS is the same in the two paths. Any confederation sub-ASs are ignored.

          In other words, MEDs are compared only if the first AS in the AS_SEQUENCE is the same for multiple paths. Any preceding AS_CONFED_SEQUENCE is ignored.

          If bgp always-compare-med is enabled, MEDs are compared for all paths.

          You must disable this option over the entire AS. Otherwise, routing loops can occur.

          If bgp bestpath med-confed is enabled, MEDs are compared for all paths that consist only of AS_CONFED_SEQUENCE.

          These paths originated within the local confederation.

          THE MED of paths that are received from a neighbor with a MED of 4,294,967,295 is changed before insertion into the BGP table. The MED changes to to 4,294,967,294.

          THE MED of paths that are received from a neighbor with a MED of 4,294,967,295 are considered valid and are inserted into BGP table with effect to Codes fixed for Cisco bug ID CSCef34800.

          Paths received with no MED are assigned a MED of 0, unless you have enabled bgp bestpath med missing-as-worst .

          If you have enabled bgp bestpath med missing-as-worst , the paths are assigned a MED of 4,294,967,294.

          If you have enabled bgp bestpath med missing-as-worst , the paths are assigned a MED of 4,294,967,295 with effect to Codes fixed for Cisco bug ID CSCef34800.

          The bgp deterministic-med command can also influence this step.

          Refer to How BGP Routers Use the Multi-Exit Discriminator for Best Path Selection for a demonstration.

          Prefer eBGP over iBGP paths.

          If bestpath is selected, go to Step 9 (multipath).

          Note: Paths that contain AS_CONFED_SEQUENCE and AS_CONFED_SET are local to the confederation. Therefore, these paths are treated as internal paths. There is no distinction between Confederation External and Confederation Internal.

          Prefer the path with the lowest IGP metric to the BGP next hop.

          Continue, even if bestpath is already selected.

          Determine if multiple paths require installation in the routing table for BGP Multipath.

          Continue, if bestpath is not yet selected.

          When both paths are external, prefer the path that was received first (the oldest one).

          This step minimizes route-flap because a newer path does not displace an older one, even if the newer path would be the preferred route based on the next decision criteria (Steps 11, 12, and 13).

          Skip this step if any of these items is true:

          You have enabled the bgp best path compare-routerid command.

          Note: Cisco IOS Software Releases 12.0.11S, 12.0.11SC, 12.0.11S3, 12.1.3, 12.1.3AA, 12.1.3.T, and 12.1.3.E introduced this command.

          The router ID is the same for multiple paths because the routes were received from the same router.

          There is no current best path.

          The current best path can be lost when, for example, the neighbor that offers the path goes down.

          Prefer the route that comes from the BGP router with the lowest router ID.

          The router ID is the highest IP address on the router, with preference given to loopback addresses. Also, you can use the bgp router-id command to manually set the router ID.

          Note: If a path contains route reflector (RR) attributes, the originator ID is substituted for the router ID in the path selection process.

          If the originator or router ID is the same for multiple paths, prefer the path with the minimum cluster list length.

          This is only present in BGP RR environments. It allows clients to peer with RRs or clients in other clusters. In this scenario, the client must be aware of the RR-specific BGP attribute.

          Prefer the path that comes from the lowest neighbor address.

          So no, it can be be neither the shortest or the fastest if you want it too.

        2. Anonymous Coward
          Anonymous Coward

          Re: not so clear

          And now for how Juniper does it:

          Understanding BGP Path Selection

          For each prefix in the routing table, the routing protocol process selects a single best path. After the best path is selected, the route is installed in the routing table. The best path becomes the active route if the same prefix is not learned by a protocol with a lower (more preferred) global preference value. The algorithm for determining the active route is as follows:

          Verify that the next hop can be resolved.

          Choose the path with the lowest preference value (routing protocol process preference).

          Routes that are not eligible to be used for forwarding (for example, because they were rejected by routing policy or because a next hop is inaccessible) have a preference of –1 and are never chosen.

          For BGP, prefer the path with higher local preference.

          For non-BGP paths, choose the path with the lowest preference2 value.

          For BGP, prefer the path with the shortest autonomous system (AS) path value (skipped if the as-path-ignore statement is configured).

          A confederation segment (sequence or set) has a path length of 0. An AS set has a path length of 1.

          For BGP, prefer the route with the lower origin code.

          Routes learned from an interior gateway protocol (IGP) have a lower origin code than those learned from an exterior gateway protocol (EGP), and both have lower origin codes than incomplete routes (routes whose origin is unknown).

          For BGP, prefer the path with the lowest multiple exit discriminator (MED) metric.

          Depending on whether nondeterministic routing table path selection behavior is configured, there are two possible cases:

          If nondeterministic routing table path selection behavior is not configured (that is, if the path-selection cisco-nondeterministic statement is not included in the BGP configuration), for paths with the same neighboring AS numbers at the front of the AS path, prefer the path with the lowest MED metric. To always compare MEDs whether or not the peer ASs of the compared routes are the same, include the path-selection always-compare-med statement.

          If nondeterministic routing table path selection behavior is configured (that is, the path-selection cisco-nondeterministic statement is included in the BGP configuration), prefer the path with the lowest MED metric.

          Confederations are not considered when determining neighboring ASs. A missing MED metric is treated as if a MED were present but zero.

          Note: MED comparison works for single path selection within an AS (when the route does not include an AS path), though this usage Is uncommon.

          Prefer strictly internal paths, which include IGP routes and locally generated routes (static, direct, local, and so forth).

          Prefer strictly external BGP (EBGP) paths over external paths learned through internal BGP (IBGP) sessions.

          For BGP, prefer the path whose next hop is resolved through the IGP route with the lowest metric.

          Note: A path is considered a BGP equal-cost path (and will be used for forwarding) if a tie-break is performed after the previous step. All paths with the same neighboring AS, learned by a multipath-enabled BGP neighbor, are considered.

          BGP multipath does not apply to paths that share the same MED-plus-IGP cost yet differ in IGP cost. Multipath path selection is based on the IGP cost metric, even if two paths have the same MED-plus-IGP cost.

          For BGP, if both paths are external, prefer the currently active path to minimize route-flapping. This rule is not used if:

          path-selection external-router-id is configured.

          Both peers have the same router ID.

          Either peer is a confederation peer.

          Neither path is the current active path.

          For BGP, prefer the path from the peer with the lowest router ID. For any path with an originator ID attribute, substitute the originator ID for the router ID during router ID comparison.

          For BGP, prefer the path with the shortest cluster list length. The length is 0 for no list.

          For BGP, prefer the path from the peer with the lowest peer IP address.

          By default, only the multiple exit discriminators (MEDs) of routes that have the same peer autonomous systems (ASs) are compared. You can configure routing table path selection options to obtain different behaviors.

          The third step of the algorithm, by default, evaluates the length of the AS path and determines the active path. You can configure an option that enables Junos OS to skip this third step of the algorithm by including the as-path-ignore option.

          Note: The as-path-ignore option is not supported for routing instances.

          To configure routing table path selection behavior, include the path-selection statement:

          path-selection {

          (always-compare-med | cisco-non-deterministic | external-router-id);

          as-path-ignore;

          med-plus-igp {

          igp-multiplier number;

          med-multiplier number;

          }

          }

          For a list of hierarchy levels at which you can include this statement, see the statement summary section for this statement.

          Routing table path selection can be configured in one of the following ways:

          Using the same nondeterministic behavior as does the Cisco IOS software (cisco-non-deterministic). This behavior has two effects:

          The active path is always first. All nonactive but eligible paths follow the active path and are maintained in the order in which they were received, with the most recent path first. Ineligible paths remain at the end of the list.

          When a new path is added to the routing table, path comparisons are made without removing from consideration those paths that should never be selected because those paths lose the MED tie-breaking rule.

          Note: The result of these two effects is that the system only sometimes compares the MED values between paths that it should otherwise compare. Because of this, we recommend that you not configure nondeterministic behavior.

          Always comparing MEDs whether or not the peer ASs of the compared routes are the same (always-compare-med).

          Comparing the router ID between external BGP paths to determine the active path (external-router-id). By default, router ID comparison is not performed if one of the external paths is active. You can force the router ID comparison by restarting the routing process with the restart routing operational-mode command.

          Adding the IGP cost to the next-hop destination to the MED value before comparing MED values for path selection.

          BGP multipath does not apply to paths that share the same MED-plus-IGP cost, yet differ in IGP cost. Multipath path selection is based on the IGP cost metric, even if two paths have the same MED-plus-IGP cost.

          This address is the IP address that is used in the BGP neighbor configuration. The address corresponds to the remote peer that is used in the TCP connection with the local router.

        3. This post has been deleted by its author

      2. Anonymous Coward
        Anonymous Coward

        @ac not so clear

        Entirely correct. This just proves that the default route is congested. Unfortunately people don't like to think and just go OHHH that looks like it contradicts the position most convenient to me, throw him to the lions because its easier than thinking.

        The guy in the video is basically forcing the route out over another asn that isn't congested. That doesn't exonerate or implicate verizon, it just shows that the default route is congested. What would be interesting to see is if that 375kbps varies over a week. If its 375kbps at 4am Tuesday morning and 375kbps on Thursday evening prime time then that is suspect.

        This real issue here is simply, can Verizon use its position as both a backbone, a tv provider and a last mile provider to force netflix to pay it money. Last mile networks take money to maintain as do backbones. Put simply If verizon let other companies peer to its network with a greater disparity in data ingress and egress than Netflix then they are manipulating the market, if they are treating netflix the same as any other company that wants to connect to their network than they are not.

        If you want to change the situation then you need to understand it and analyse what is going on. Hit them with some real evidence. Show the above to Verizon they will shurg and say yeah, the link is congested because they hit their contractual ratio via peered links and saturated their paid link.

        1. Sir Runcible Spoon

          Re: @ac not so clear

          If Netflix ran a VPN terminator that this guy could connect to that used the same path through Verizon's network then that might be a useful comparison to make.

    4. Don Jefe

      You've got to think bigger than 'anti-competitive behavior designed to influence the service selection choices of consumers'. While I'm certain the flow chart of that plan must be gloriously complex, the plan has the, fairly common, flaw in pseudo-conspiracies. In this case it would be Verizon who has implemented a strategy that has only one desirable outcome (subscribers dropping Netflix in favor of Verizon content) and failed to control the space between themselves and the target (subscribers).

      You want the targets of your conspiracy to act/respond/be impacted in a manner you desire. Right? The general rule in designing is as follows:

      The accuracy with which you can dictate the actions of your target is directly proportional to the amount of control you exercise over any alternative actions the target can take. Therefore, a proper conspiracy will have a variety of desirable potential outcomes in proportion to the variety of options the target has.

      In this case, there's a shitload of actions consumers can be expected to take before canceling one service and subscribing to another, more expensive service that has different content they obviously didn't feel the need to buy anyway. There's simply no way a large company is going to risk Justice Department investigation to create a conspiracy that has only one desirable outcome. That's just amateurish in the extreme. Verizon are dicks, not stupid.

      What you're actually seeing amounts to a border skirmish between two entities gearing up for full scale war on pending net neutrality legislation. At this very moment both sides are using the exact same information to serve as proof regulated net is bad, or good, depending in which side you're on. While public lobbying isn't as much fun as a conspiracy, in this case public lobbying is what we've got.

      1. h4rm0ny

        >>"Verizon are dicks, not stupid"

        I'm not full convinced of that. Anyone else remember the person who recorded their entire phone call with Verizon where they tried to explain decimal points to a succession of Verizon employees? Without success, by the way.

        1. Don Jefe

          You're talking about Verizon staff, not executive officers. You might as well compare and contrast a stoat and the front bumper of a 1966 GMC pickup truck.

          This simply isn't general staff stuff. They don't develop plans to degrade service for users of one of the most popular streaming services in the US and piss those users off. Piss of millions of the customers of a service that is one of the most bandwidth intensive things on Earth. To do it all just as one of the largest, most focused and expensive lobbying offensives in the last decade is gearing up is most certainly not within the purview or capabilities of general staff.

          What I was trying for with my earlier comment was to illuminate the fact that this issue isn't a simple thing general staff or some particularly Goldbergian plot by an imbalanced VP of Sales. This is all 100% the pre-war establishment of the theater of conflict. Part of that process is to influence the civilians within the borders of the theater. In any large conflict your most valuable assets are the civilians who get caught between the two opposing forces. You either get them on your side or foment discord and confusion sufficient to neutralize those not supporting you.

          You watch how this plays out. I've been at this far too long, and done far to well at it, to not recognize this for what it is. In the very short term I guaranfuckingtee you'll see this positioned as proof that 'consumers are being harmed because a few wealthy companies are monopolizing available bandwidth and expecting consumers to fund infrastructure expansion when it is clearly the companies sucking up all the bandwidth who should be paying for that enhanced infrastructure'.

          Netflix will position this as proof that 'customers and businesses will be harmed if ISP's are allowed to prioritize traffic on an individual basis. If subscribers are receiving suboptimal service from the ISP's that an issue the ISP's need to address internally'.

          The people who apparently didn't like my earlier comment are confusing the issues here. Absolutely none of this has a damn thing to do with consumers. The consumers are pawns in a battle between completely different entities. Verizon wants their customer (Netflix) to pay more for access into the consumers homes and Netflix wants their vendor (Verizon) to maintain their current billing model.

          There's no doubt the consumer is going to get fucked, one way or another, but that's collateral damage as far as the ISP's are concerned. It's just playing into the hands of the ISP's if the consumers believe they are the primary target of the ISP's machinations in this affair. That's in no way intended as a swipe at the general consumer, not at all. But the general consumer is simply no match for the resources and complexity of manipulation brought to bear when large companies really go to war. They think they understand what is happening, but that's the point. They're looking the wrong way when the big company comes at them sideways. Just because a company isn't responsive and agile when addressing customer complaints/issues it's foolish to think those companies can't be responsive, agile and cunning where their interests are concerned.

    5. Naughtyhorse

      Duhhh!

      What's the point in buying a government if you can't then go on to recoup your costs by ripping off the public.

      I think it's called 'the market' :-)

      Before making a right wing knee-jerk DV, think about it. what is the exact mechanism that results in this insanity? the market. it might be broken, but nonetheless that's what it is. the power resides with the politicians, being a politician is heinously expensive, so if you want to continue to be a politician you gotta suck a lot of cocks. Corporations have a shitload of cash which they (wisely) invest in buying the government, who in turn ensure the corp is still there writing cheques at the next electoral cycle.

      supply and demand, just in this instance (and lets be honest in just about every other conceivable instance) the people get screwed along the way.

      democracy don't mean shit if collectively you are going to consistently vote like twats.

      simples

  2. Shadow Systems

    Raise your hand if surprised...

    *Cricket chirping*

    *Tumbleweed rolls past*

    *Cheesy "Ghost Town" music*

    Anyone? Anyone? Beuller? Beuller?

    *Cough*

    It was the exact same situation with Comcast strangling Netflix traffic. Use your raw Comcast bandwidth & the video won't play worth crap, but open a VPN to Netflix and !BAM! you were watching the full HiDef stream without buffering.

    Verizon strangling Netflix traffic, claiming it's everyone *BUT* Verizon's fault, yet if you visit Netflix via your raw Verizon bandwidth the video plays like crap, and that VPN unleashes the full stream without buffering.

    Even the *BLIND GUY* can see that Verizon (and Comcast) is full of utter shite, and deserves to be taken out & shot for the lie.

    I'm paying for $X bandwidth. It doesn't matter where I choose to go, you'd damn well better deliver AT LEAST that level of bandwidth. If the other server can't send it that fast, fine, but don't tell me it's the other server's fault when a simple VPN over the same connection proves you lying out your arse. Give me my fekkin bandwidth that I pay for, or don't be surprised when I decide to turn into an Insane Clown & run you over repeatedly with a MiniCooper. I'll charge your Next Of Kin extra for the Cackling & Cleaning.

    1. Down not across

      Re: Raise your hand if surprised...

      FCC is collecting opinions until Sep 10th I believe.

      The FCC's draft rules propose banning ISPs from blocking users' access to websites or applications but allowing some "commercially reasonable" deals between content providers and ISPs to prioritize delivery of some web traffic.

      Is throttling same as blocking. In users view it no doubt is. Whether FCC thinks so is another matter. Guess they'll do that as "prioritizing" traffic.

      FCC should just reclassify as telecommunications service, which is something AT&T, Comcast and Verizon are vehemently opposing. That would kill their planned cash cow.

  3. Anonymous Coward
    Flame

    A glimpse of the future...

    This is what Verizon wants the future to be, pay them their danegeld, or they slow you down to a crawl.

    1. Down not across

      Re: A glimpse of the future...

      This is what Verizon wants the future to be, pay them their danegeld, or they slow you down to a crawl.

      So do AT&T and Comcast. Easy to see why, they are looking to cash in on the big content providers. Not to mention there are conflicts of interest with their own content services. As usual it is all about money.

      Net neutrality is important. The best/easiest would be FCC reclassification, but I'm not convinced they have the cajonas to do it. Wheeler threw the threat of reclassification out there and now is backpedaling so fast the dynamo on the bike generates -48V

  4. Dave 52

    This is why enforcing net neutrality is important

  5. Jamie Jones Silver badge

    Probably, but not necessarily ...

    Not defending Verizon, who should not be dragging their feet over this issue, but this VPN situation could occur 'accidentally' if the guys VPN goes via another route not usually congested with Netflix traffic.

    Edit:

    Just checked his blog. He basically said this himself:

    My hypothesis here was that by connecting to a VPN, my traffic might end up getting routed through uncongested tubes. Basically, if Verizon is not upgrading the tubes that go to Netflix, maybe I can connect to a different location (via VPN) first where Verizon will have good performance and there will be no congestion between location 2 and and Netflix.

    .... but then later seems to forget that this could be incidental rather than intentional throttling.

    1. Down not across

      Re: Probably, but not necessarily ...

      Not defending Verizon, who should not be dragging their feet over this issue, but this VPN situation could occur 'accidentally' if the guys VPN goes via another route not usually congested with Netflix traffic.

      Quite. Which is what I tried to point out in some posts. So far all parties have blamed another party as is quite common in corporate world. Let's face it at least one of the parties is lying, or not telling the whole truth anyway. In order to be objective (yeah this is the Reg, what am I thinking!) I was considering options that might explain the situation.

      Obviously not commenting on allegations does not help as it is easily interpreted as "damn, got caught. Can't talk out of this one". Of course considering Verizon's stance on net neutrality and lobbying against it is another nail in this coffin.

      If anything this is good example what is likely to happen should the opponents to net neutrality win. First its some high bandwidth entertainment. Anyone think it'll stop there? No, didn't think so either.

    2. Anonymous Coward
      Anonymous Coward

      Re: Probably, but not necessarily ...

      The route the traffic takes it not a factor and here is why. It is Verizon that has stated the congestion is not in the Verizon network, but the companies that Netflix uses. If the link that is congested is from Level-3 to Verizon, then the congestion is right at the front door on Verizon itself which is why Level-3 has stated that they are willing to hook a few more cables up.

      Verizon would counter argue in that they are only responsible for the uplink portion and not the downlink. Which is fine, but Verizon doesn't want to allow Level-3 the opportunity to fix the portion they are responsible for either.

      Maybe the extra 10 to 20 GBps that Level-3 wants to add will cause issues for Verizon on their infrastructure is another possibility as Verizon knows that extra traffic from Level-3 would quickly be put to use.

      1. Jamie Jones Silver badge

        Re: Probably, but not necessarily ...

        " The route the traffic takes it not a factor and here is why. It is Verizon that has stated the congestion is not in the Verizon network, but the companies that Netflix uses. If the link that is congested is from Level-3 to Verizon, then the congestion is right at the front door on Verizon itself which is why Level-3 has stated that they are willing to hook a few more cables up. "

        No. As I said, not necessarily.

        It is established that the main route for Netflix to Verizon is via level3.

        They may, however have a link via (say) Hurricane Electric which happens to be the route used by the ISP the VPN runs under.

        If this link isn't saturated, and the VPNs ISPs link to Netflix isn't saturated, then he will get the faster speeds as he will no longer be using the Level3 link. (netflix -> vpn -> Hurricane Electric -> Verizon) without it meaning that Verizon is specifically throttling Netflix.

        If he names his VPN provider and/or provides traceroutes, it will be clear. If his VPN also goes via the same Level3 route, then, yes, throttling is being intentionally done. If not, we simply can't be sure based on the evidence so far provided.

        1. Anonymous Coward
          Anonymous Coward

          Re: Probably, but not necessarily ...

          We already know the links to Level-3 are congested. Back in May there was an article that stated some names but not all. AT&T, Comcast and Time Warner were mentioned but it was just in that Level-3 peers with 51 other companies like themselves and to companies that offer consumer broadband service. It went on to say that 39 of those companies, the links run at 36% or less. We know that Verizon also peers with Level-3 but they were not mentioned in that article. Ok, so that leaves 12 companies left and Level-3 said that of those 12, 6 are making plans to increase the bandwidth between it and Level-3. Once again, Verizon was still not mentioned. That leaves 6 left and Level-3 stated that one of those companies is in Europe, so that leaves 5. Those 5 are refusing to allow Level-3 to add capacity to them and the current links are at least 90% utilized. Verizon was not mentioned in that article, but guess what, Level-3 has mentioned that in this latest article. Level-3 stated that they have free ports available and want to increase the capacity to Verizon but Verizon won't let them. So, it doesn't matter what a traceroute shows as we already know there is congestion as that is why Level-3 wants to add capacity to Verizon. So, it is Verizon and Verizon alone that is the cause of this issue and they have no interest in getting it resolved. The reason is simple, they are trying to force Netflix to buy access from Verizon. Verizon has even stated that companies like Netflix should not be using peers to transmit their data, but to go directly to Verizon. The companies that peer with Verizon (like Level-3) should be very concerned at what Verizon is doing; they are trying to poach the customers of the peers. The peers should force the traffic over the public peering points and cause issues for Verizon. So if Verizon thinks that the peering links should be symmetric, then the peers should make sure it is 1:1 and the excess goes over the public peering points. This would congest the public peering points that Verizon has and would not only impact the broadband customers, but the enterprise customers and their mobile customers as well. They would b crying murder and Verizon would fail to meet the QoS promised. All it would take is an hour or two before Verizon would think twice of the game they are playing here. Their peers couldn't be held liable as they were just following the agreement; 1:1 traffic over the private peering links. If Verizon doesn't have enough capacity on the public peering side, that is not the fault of anyone but Verizon itself.

          1. Jamie Jones Silver badge
            Facepalm

            Re: Probably, but not necessarily ...

            Cheers for the downvote.

            Your reply says nothing to contradict what I wrote.

            Either you have comprehension difficulties, or you just don't understand.

            I said nothing to suggest that their link to Level 3 wasn't a problem. I was explaining that the fact he gets faster access via a VPN doesn't necessarily mean Verizon are specifically throttling Netflix packets.

            And yes, traceroutes would be relevant to help determine this.

            Please come back when you've acquired a clue.

            1. Sir Runcible Spoon

              Re: Probably, but not necessarily ...

              Unless Verizon are throttling all traffic to the Netflix network via L3 it should be possible to perform some other tests that create a like for like bandwidth comparison.

              1. Run raw Netflix feed through a VPN to Netflix

              2. FTP some large file from Netflix

              3. SCP a large file from Netflix

              4. FTP over HTTPS/HTTP

              These tests could also be performed using a variety of ports. If Netflix were serious about proving this they would provide test servers and instructions for their users to compare the test traffic to their Netflix raw feed. This should reveal the full extent of any throttling that wasn't congestion related.

              1. Jamie Jones Silver badge
                Happy

                Re: Probably, but not necessarily ...

                Yep!! Good ideas!

                Actually, you could probably do it just by testing to somewhere that also peers through the same L3 link, just in case Verizon were allegedly throttling everything to the Netflix addresses.

                I'd love a shell account on a Verizon connected host right now!

    3. Tom 13

      Re: Probably, but not necessarily ...

      You're down in the weeds on this. Come up to a higher level.

      Verzion has the pipes to send the data to their customers. They aren't doing it. Therefore it is Verizon's issue, not Netflix. I know what I pay Verizon a month, and I know what I pay Netflix a month. I know what Verizon promised me, I know what Netflix promised me. Verizon is the one who isn't delivering and I'm paying them better than 5 times as much money.

      1. Jamie Jones Silver badge

        Re: Probably, but not necessarily ...

        "You're down in the weeds on this. Come up to a higher level."
        Really?
        "Verzion has the pipes to send the data to their customers."
        I agree.
        "They aren't doing it."
        I agree.
        "Therefore it is Verizon's issue, not Netflix."
        I agree.
        "I know what I pay Verizon a month, and I know what I pay Netflix a month."
        I agree.
        "I know what Verizon promised me, I know what Netflix promised me."
        I agree.
        "Verizon is the one who isn't delivering and I'm paying them better than 5 times as much money."
        I agree.

        Now, how does this contradict the *only* point I've made (many times) in response to this article, which is that nothing in the original blog post proves specific netflix throttling?

        Is there some comprehension issue here?

        Throttling doesn't mean "We know there are problems with the Netflix connection, but are denying it and doing bugger-all about it"

        It means: "Oh look, that packet is from netflix. Let's intentionally artificially slow down it's delivery."

  6. I. Aproveofitspendingonspecificprojects
    Paris Hilton

    2008?

    This "cure" was pointed out in an article written in 2008 -after it was covered in a computer show?

    And it isn't the front end of all Linux operating systems yet?

    WTFhIT been F hiding for F 6 F years!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  7. Gene Cash Silver badge

    This happens to me as well on Time Warner/Brighthouse.

    I get 1.5mb/sec capped normally. You can see the straight-line limit on xosview. It's pretty obvious it's capped. Bit-torrent is capped to the point of unusability, something like 0.25mb/sec.

    If I use VPN, it more than doubles to almost 4mb/sec for everything. Youtube, NetFlix, http, bit-torrent, etc.

    I can't see how it can go through another "uncongested route" as it still has to make it out of my ISP.

    1. Down not across

      I can't see how it can go through another "uncongested route" as it still has to make it out of my ISP.

      Because due to peering and transit agreements there are often more than one route of the ISP.

      Your VPN endpoint is unlikely to be your final target (that you were getting to without VPN) and hence could end up routed differently.

      So you could easily end up with a route, with more hops, that is ungongested and with better throughput.

      Now this is not to say ISPs don't cap things. Many cap stuff like BT etc. Some are upfront about it. some are not.

      As you pointed out it can be fairly obious if it is a case of limit/throttling rather than natural congestion.

      1. Anonymous Coward
        Anonymous Coward

        Um, so how does that work then?

        Many of us do not care how the "accident" happens. You can be as technical as you like, but the end result is the problem.

        What if we pay for a cab to go to work and by "accident" they take the longest slowest route? If we see right next to us in plain day another road clear, we are going to be FUMING.

        Likewise, if an ISP conveniently never changes their routes, what do we suspect?

        We are not talking about paid for links here are we? If the packet is encrypted, how come that packet gets preferential treatment? The encrypted packed gets no bonus from peering deals... so why should a Netflix packet suffer WORSE with no peering deals? That's the fishy elephant in the room!

        1. Jamie Jones Silver badge

          Re: Um, so how does that work then?

          No-one is defending Netflix - we're just pointing out that it's *not necessarily* specific throttling. And, it has nothing to do with encrypted packets - just that someone is bouncing via an alternate route.

        2. Anonymous Coward
          Anonymous Coward

          Re: Um, so how does that work then?

          Ok. So it's not just the congestion then? If we assume the VPN has both quicker networks and better links than Verizon. Which still means the slow network is with Verizon.

          I've seen peering/routing problems with my own ISP through bugged routing tables. Usually fixed quickly though. Examples are routing content across the Atlantic and back again, or across Europe and back, because it's stuck thinking it needs to go through something special (NSA? /sarc).

          If my ISP does that, I blame the ISP, not the website. Still want to blame everyone but Verizon here?

          1. Jamie Jones Silver badge
            Facepalm

            Re: Um, so how does that work then?

            Can you read?

            1) I never said there wasn't congestion.

            2) I never blamed anyone else for the problem.

            3) Nothing to do with routing. Unfortunately, capitalism get's in the way of your blinkered utopian view on how the internet works. You can't blindly reroute via every available route - some links are contracted for specific routes only.

            An actual real-case example.

            I used to work for a large International tech company. We had fast internet connectivity, although it was often unreliable.

            At one stage we bought out another company, and as their network already used an IP address range that didn't clash with ours, their network was soon fully absorbed and routable to/from ours.

            For the time being at least, they continued to run as a separate company with their own budget and management structure and accounts.

            They also had their own internet link, which although wasn't as beefy as ours, was far more reliable.

            Technically , our network could be configured to route to the internet via their link (the non-private addresses that didn't need NAT, at least)

            Indeed, if our link was down, some of us would bounce our SSH connections through them, but there would have been hell to play if the main connections automatically routed to the internet this way.

            As I (and others) have continually repeated in this thread, the issue was that from the evidence given, YOU CAN NOT DEDUCE THAT VERIZON ARE SPECIFICALLY THROTTLING PACKETS FROM NETFLIX.

            That is the ONLY thing we've said on the matter, and can''t be said any clearer.

            You are therefore either trolling, or incredibly stupid - stupid of the worst kind - the kind who blindly believes they understand something which is actually totally beyond them; the type of person who ends up making monumental cockups because 'they know best'.

            Not understanding something is fine - we can't all be experts in every field. What defines the stupid like you is you don't have a clue, but think you do. You then proceed to embarrass yourself with stupid postings - at least, you would be embarrassed if you weren't living in your stupid fantasy ego created world in which there is any merit in your anal dribblings.

            Or maybe you *are* beginning to realise, which is why you're posting anonymously. If that is the case, then congratulations! Baby steps etc.

            If not, don't worry there kiddo, you'll be perfect for management.

  8. This post has been deleted by its author

  9. Christian Berger

    It's similar with Deutsche Telekom here

    They don't do peering, so if you want to connect to them, they'll charge you as if you were their only upstream ISP, and they charge about double of what the competition does. Therefore, as far as I know, they aren't connected to Google or any of the large Internet exchanges.

    However local hosting providers typically are connected to them as well as to the nearest Internet Exchange. So routing through your server at such a hosting company can make your Youtube work considerably better.

  10. Mark #255

    time for Netflix to test things properly

    So evidently, what is needed is for Netflix to quietly set up some VPNs of their own, and lend them to complaining customers, just to see where the problem is. (Of course, this armchair engineer leaves it as an exercise for the reader to ensure that/maximise the chance that the bits do get shunted down the correct pipes so it's a true test.)

    1. phil dude
      Thumb Up

      Re: time for Netflix to test things properly

      this sounds spot on, and I think they are already doing this. The google play store update netflix a few times, but I read in the blurb that it would be collecting bandwidth stats.

      I would think that so long as it doesn't break the streaming, netflix could collect a great deal of data, and then perhaps a lawsuit would settle this...

      I don't see how deliberately not providing bandwidth you paid for, is not a breach of contract?

      P.

    2. Anonymous Coward
      Anonymous Coward

      Re: time for Netflix to test things properly

      No, what they should do, get an extra block of addresses from Level-3 or somewhere else, encrypt the packets and turn the service on. Chances are, whatever method Verizon is using to limit, would not work in this case and users would get normal stream bitrates until Verizon caught on and limited it. This would be enough to show that it is Verizon doing it on purpose; it worked and then stopped all while the original service still didn't work properly at all.

  11. Efros

    I've seen something similar with Time Warner Cable and believe it or not Youtube.

  12. TJ1

    Why aren't customer's sueing? Business units, transit, and peering

    This being America why aren't we hearing of law suits - maybe class actions - by Verizon retail customers due to a breach of the the contract between customers and Verizon? After all, the customers presumably pay for unlimited access (both in reach and content) up to their connection speed.

    Is it that the contract doesn't commit Verizon to providing the headline service advertised (Tx/Rx speeds) and allows for Verizon to force contention on its retail customers ?

    Reading the 2013 annual report shows that the business unit responsible is Verizon Wireline. Within that they have "Mass Market" and "Global Enterprise" units. Verizon operates several business units, one of which is the consumer 'last-mile' ISP the customers contract with which is in the "Mass Market" unit. This Verizon Consumer ISP then contracts with the "Global Enterprise" Verizon Enterprise entity that sells transit connectivity and capacity to business customers [1].

    Verizon Enterprise operates on the same basis as other transit and connectivity providers with a combination of public peering and private peering. In the private peering there are several types of contract and, as I understand it, the contract between Verizon Enterprise and Level 3 (in terms of these congested connections affecting Netflix) is a "balanced flow" agreement.

    Balanced flow agreements are no-cost contracts between connectivity providers that expect each party to be sending the other roughly the same amount of traffic. To avoid the administrative overhead of metering and invoicing each other when the net effect is basically $0, they agree to peer with each other without charge.

    The Verizon Enterprise Peering Agreement [1] specifically addresses this issue at:

    "1.2 Traffic Exchange Ratio. The ratio of the aggregate amount of traffic exchanged between the Requester and the Verizon Business Internet Network with which it seeks to interconnect shall be roughly balanced and shall not exceed 1.8:1."

    So for flows whose aggregate ratio exceeds 1.8:1 there is usually an invoice due payable by the party sending the most data on the link.

    It *seems* as if the cruz of the Netflix issue is that they have contracted with Level 3 to provide their transit connectivity from the Netflix data-centres.

    Level 3 are then routing that traffic through their cheapest routes to their Verizon Enterprise peering points.

    The flows on these connections are now overwhelmingly Level 3 > Verizon and so Verizon Enterprise probably sees it as an abuse of the balanced flow agreements. If you set aside the emotion surrounding Netflix then this makes perfect sense and is the normal course of business for transit.

    Presumably Level 3 priced their Netflix contract on being able to use their Verizon Enterprise peering points at no cost and are put-out that they may have to start paying and would prefer that not to happen.

    Netflix are caught out through no fault of their own since their business model is probably costed on the basis of the cost of Level 3 transit and so place the blame on the amorphous Verizon Communications group.

    [1] http://www.verizonenterprise.com/terms/peering/

    1. Anonymous Coward
      Anonymous Coward

      Re: Why aren't customer's sueing? Business units, transit, and peering

      While the peering arrangement is the issue I have always had issues with them when it comes to consumer broadband.

      1) Consumer broadband is typically asymmetric; they have more download speed than upload.

      2) It is not Level-3 or even Netflix blindly sending traffic; it is the broadband customer requesting it. if anything, the peering should be reversed from how it is today.

      3) What is the broadband customer paying for then? They offer different speed packages to make more money, they put bandwidth caps in place and charge you if you go over. You are already paying for the right to access Netflix and get the speed you are paying for all while staying under your cap.

      Verizon appears to be the worst when it comes to the peering process in that they want every company to buy links from them directly and not do peering. Maybe companies like Level-3, Hurricane Electric, Amazon, etc. all should on the same day disable all private peering with Verizon and force the traffic though public peering points or at the very least, try to keep a 1:1 ratio on the private peering links with the rest going over the public peering. This would quickly saturate the public peering links that Verizon has and would impact not only the consumer broadband customers, but the Verizon business customers as well. It is the business customers that Verizon cares about as they are paying hefty service fees. That might be enough for Verizon to change their tune as they would want the private peering links to be used again. Hurt the Verizon business model should be what the peers are after.

      1. Tom 13

        Re: Verizon appears to be the worst

        Agreed enough to give you an upvote, but this bit isn't correct. They're pretty much all about the same. We're currently with Verizon because after the conversion to high def, Comcast had their cable lines so screwed up they were fogging the picture on their cable package. Initially I didn't go with them because they were known to be virus central on shared connections. FIOS and DSL avoided the shared line issues.

    2. Tom 13

      Re: Why aren't customer's sueing? Business units, transit, and peering

      In my case it's because the Verizon bill is technically in my landlord's name, so I don't have standing to sue. Even if it were, without being certified as a class, I'd have to go to small claims court where I'd have to pony up the cash to fight their well paid lawyers. So I'd still be out of luck.

  13. nematoad

    I should be so lucky!

    "Nederkoorn has posted a video that first shows a Netflix stream running at 375kbps on what he says is his regular internet connection and configuration. He then connects through a VPN service and reloads the video, which promptly increases to speeds of up to 3,000kbps."

    375kbps? Good God if I got that speed I'd think that I had died and gone to heaven.

    Still we have that nice Mr. Cameron's word for it that most of the denizens of these sceptic isles will soon get "superfast" broadband of up to 2 Mbps.

    Can't wait.

  14. robe.rabun

    Netflix VS Redbox Instant

    Has anyone using Verizon run a side by side comparison of Netflix up against Verizon's Redbox Instant streaming service?

    Would be interesting to see how much faster their own service would be compared to Netflix.

    Comcast is NO BETTER btw.

  15. James Finnie

    Could also be down to Verizon prioritising VPN traffic above all other traffic; which is a sensible thing to do as VPN connections don't like poor performing links.

    1. Anonymous Coward
      Anonymous Coward

      They are not prioritizing VPN traffic; there is no incentive for them to do that. Verizon wants the ability to be able to charge to do that though; so why would they do it for free?

  16. DrXym

    This is why net neutrality is so important

    Without net neutrality, ISPs will start throttling services and holding them to ransom which essentially means people pay more to use them. Aside from that people find themselves involuntarily segregated into service packages where one ISP works well with one set of services and another ISP works well with another set of services.

    The only reason I see to break net neutrality is if the ISP actually paid Netflix or some other service to install servers on their backbone to minimize latency.

    In other words, the ISPs should not be allowed to throttle or impede access to service outside of their network but they should be able to add proxies or servers inside their network to improve service..

  17. Mikel

    John Gilmore

    John Gilmore famously said that the Internet is designed to route around damage, and censorship is damage. Here is an Internet user interpreting a network congestion point as damage and routing around the damage using a VPN. These stories have been around for years and ultimately someone will come up with a distributed VPN router that will eliminate this problem for everyone all at once. For now though, as we often have had to do while we wait for these cures to come online, you can fix it yourself manually as this fellow has done.

  18. W. Anderson

    More proof of large, greedy corporate dominance

    It is not surprising that Verizon would stoop to such low level of integrity in throttling service to Netflix customers jusst to suit their commercial gains for charging extra for bandwidth which should be minimal.

    However it still remains for all the companies serviced by Netflix like Level 3, Nederkorn's ISP and others which have their customers severely negatively impacted by Netflix's actions will band together and post a strong, formal objection to the FCC, probably out of fear of retaliation by Verizon and it's status of preference and special positive treatment by FCC oligarth obedient board.

    A sad state of affairs in USA, whose citizens are already receiving 2nd class Internet access at higher rates than most of the other G20 and developned nations.

    So much for the "Land of the Free", the "Greatest Nation n Earth", "American Exceptionalisn" and other crass slogans constantly spewed by draconian, greedy corporations and an ignorant, naive public.

  19. ecofeco Silver badge

    One word

    tracert

    Did he run it?

  20. jjackson

    I just saw this article and yeah it's messed up what the isps are doing. I'm glad my friend turned me on to https://ironsocket.com so I can choose between using a vpn or smart dns proxy (which is all you really need to get around the geo restrictions).

    This is the way of the world, you cannot regulate the behavior of people. If they want to watch netflix and the ISP regulates it, then the people will find a way around it to keep calm and netflix on.

  21. jameshudson

    Best VPN service for Netflix

    I highly recommend using http://www.vpnmaster.com for Netflix. It works flawless even abroad.

  22. Anonymous Coward
    Anonymous Coward

    Every ISP that tries to force subscribers to its own 'walled garden' monitised content always fails as the internet routes around.

  23. Anonymous Coward
    Anonymous Coward

    If you subscribe to a "decent" ISP

    i.e. one that has nice big fat pipes to Netflix and Youtube or even partners with them to provide hosting in their network then you won't get as much congestion will you?

    All a VPN does is virtually subscribe you to another provider - presumably in this case one that is fully performing it's job as an ISP correctly.

    TLDR: you get what you pay for.

  24. Daggerchild Silver badge

    Or, his VPN endpoint host just has better connectivity...

    A->B=1, A->C=2, C->B=2 : Therefore A->C->B=2 > A->B=1 without any 3vil filters necessary

  25. Clerk

    A VPN or DNS service can actually improve Verizon's speed regarding video streaming. I am currently using UnoTelly and I can notice the difference especially on Netflix

  26. dinaomar

    Using VPN software http://www.unlimitvpn.com/unblock-netflix/ to change your real IP address and get European IPs with connecting to different servers all over the world like US server to get American IP to be able to unblock Netflix and any other censored services

  27. kec2
    Thumb Up

    Faster Than VPN!

    I used many vpns before but VPNs have many lacks of functionality and speed. It only works with your pc mainly.

    Now I'm using Smart DNS Proxy (www.smartdnsproxy.com) and it's extremely fast. It's not like vpn but still netflix traffic routes through different route and provides much faster streaming on Verizon. The best of all Smart DNS Proxy works with all my home devices, PC, Mac, iPad, Roku, Chromecast. I love it!

    Thank You :)

  28. Alex1507

    Got it

    I can also confirm with tests conducted with a Verizon SIM and my own VPN that it prevents Netflix from being throttled.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like