back to article Lone Android dev 'almost brought down T-Mobile'

T-Mobile USA told the FCC that having an open network caused it severe overloading and network degradation, thanks to one incompetent Android developer. The developer in question created an Instant Messaging application that drove traffic up by 1,200 per cent in one instance. As it became more popular, T-Mobile was obliged to …

COMMENTS

This topic is closed for new posts.
  1. Bade

    seen that before

    "it seems likely the application was repeatedly setting up and tearing down IP connections, using each one for a very small amount of data and thus generating more signalling load than the base stations could manage."

    Isn't that EXACTLY the problem that iPhones had that caused the problems on AT&T and O2?

  2. Anonymous Coward
    Anonymous Coward

    bs

    This has nothing to do with net neutrality. Looking at wired infra that would be like saying that net neutrality forbids NOCs to counteract DDoS attacks. Net neutrality is there to protect users, if a dev writes an app that kills the network (on purpose or by accident) net neutrality should not forbid the NOC from taking action.

    Of course this is just my 2cts, the laws over in Usonia are probably being written to exactly the effect that T-Mobile is afraid of.

    1. JaitcH
      WTF?

      This is the old Lemon 4 battery extender trick

      One of the (many) complaints about Lemon 4 was dropping calls, separate and apart from it's Grip of Death problems.

      Several test facilities have attributed dropped calls to this technical slight of hand fom Apple in a feeble attempt to claim it had lengthened battery life.

      This is similar to saying a 40 watt light powered from a car battery lasts as long as an LED array from the same car battery.

      But, hey, Apple claims to have some of the best RF guys in the world ... one thing for sure they introduced unique sets of problems!

      1. ThomH

        @JaitcH

        I think there's a story on the front page about exam boards that nobody has tediously tried to position as having an Apple angle yet. You'd probably better get over there.

    2. frymaster

      actually it _IS_ saying that

      "Looking at wired infra that would be like saying that net neutrality forbids NOCs to counteract DDoS attacks"

      the way the fanatic fringe - ie the part that gets the publicity - of the net neutrality brigade are phrasing things, that's exactly what it would mean.

      I agree the circumstances in which ISPs can discriminate traffic need to be locked down, but the tubthumpers (again, only the very fringe, but the people who get quotes) are going on about total neutrality. I want ISPs to be able to:

      - deal with (D)DOSs and faulty apps like in the article

      - prioritise based on tcp/ip headers (icmp vs udp vs tcp, for example)

      - to further be able to prioritse based on traffic type (real-time VOIP and gaming vs web pages vs bittorrent/ftp/binary newsgroups)

      ...all of which make sense, but we're in danger of throwing out the quality of service baby with the sneaky underhand revenue generation bathwater

  3. Anonymous Coward
    WTF?

    The Fallacies of Distributed Computing...

    An incompetent developer is No excuse for being exempt from net neutrality.

    It seams that no one at T-Mobile have ever seen: The Fallacies of Distributed Computing.

    So here they are:

    1. The network is reliable.

    2. Latency is zero.

    3. Bandwidth is infinite.

    4. The network is secure.

    5. Topology doesn't change.

    6. There is one administrator.

    7. Transport cost is zero.

    8. The network is homogeneous.

  4. Andrew Shirley
    FAIL

    "incompetent" seems a bit harsh

    Seems to me that he used a reasonable strategy to maximise battery life and him not knowing the connection provider's overheads for different actions isn't something I'd describe as "incompetent"

    1. Rob Moir
      WTF?

      Competence

      If it isn't incompetent to develop something blindly, without thinking through the consequences of your actions for the environment, then what is it?

      1. Anonymous Coward
        Anonymous Coward

        incompetence is the new normal

        What I've seen of new programmers in the last 10 or so years is that many have no clue how a computer or network really works, nor do they care. Far too many recently "educated" programmers were brought up developing on PCs where their actions affect no one but themselves and resources are far beyond what they need for their application. Sloppy, inefficient coding is all they know. Then they get in the real world and suddenly efficiency and size matter and resources aren't free.

        They should all be forced to develop on 6809 with 16kb of RAM until they learn to stop being stupid.

      2. Cynic
        WTF?

        So you are one of these people who know everything??

        Being a dev - I can assure you that we try to learn as much as we can about the environment into which we deploy. But in the present day and age, that environment is so large and diverse that things just happen unintentionally. That is not incompetence. Lack of experience maybe - but you only get experience from making mistakes. Don't confuse the two.

        1. Semaj
          Pint

          I wish you were right

          For good devs like you and me you are correct but there are A LOT of cowboys out there who give the rest of us a bad name.

      3. Anonymous Coward
        Anonymous Coward

        Re: Competence

        > If it isn't incompetent to develop something blindly, without thinking through the consequences

        > of your actions for the environment, then what is it?

        Our companies hiring policy?

        Anon for obvious reasons....

    2. The Other Steve
      WTF?

      Shirley not

      "Seems to me that he used a reasonable strategy to maximise battery life"

      No he didn't. He used a fucking stupid strategy in a misguided attempt maximise battery life, because he didn't understand anything external to his code.

      And that's a pretty good description of incompetence.

    3. bazza Silver badge

      Fifty fifty?

      I'd estimate that the developer was partially to blame - repeated setting up / tearing down of connections is surely going to be inefficient. But what sort of OS is it that, when faced with critical network constraints like this, does nothing to stop an app from just going ahead and doing it?

      Surely then the answer is not just to fix the app. The OS is presenting an opportunity for a denial of service attack. Write a popular app, give it a command channel, then blackmail the network operators by threatening to activate the nasty payload on millions of phones that does what this app accidently did.

  5. Anonymous Coward
    FAIL

    Or T-Mobile could fix it a layer down...

    If freely sending and receiving packets is that damaging to their network, why hasn't the network operator provided a custom tcp/ip stack that queues pending packets, and bursts them on a schedule that isn't damaging to their network? Or uses some sort of heuristic that starts to queue and burst if it detects overloading, but allows quasi-real-time operation otherwise. Unless T-mobile is claiming this 'incompetent' Android developer's app hacked the hardware to send stuff over the antenna directly, presumably he's just using the standard sockets API provided. Seems to me traffic management should be done a layer or two down if it requires special attention (as evidently it does).

    1. copsewood
      Terminator

      wrong layer

      Personally I doubt T-mobile or any other operator working on their own could fix a problem in relation to technology they implement but didn't themselves develop. This seems more like a standards issue to me. Perhaps the standards developers didn't foresee this potential mobile application usage pattern. Given the difficulty and cost of transitioning from 2-3-4G and the time taken, or of industry-wide retrofitting of existing platforms, maybe this is one that will have to be patched in less optimal but more feasible places, i.e. within the applications which exercise this platform bug.

      This one isn't an issue of network neutrality as far as I am concerned, more a question of pragmatism and muddling through.

  6. bluesxman
    FAIL

    s'kiddies rejoice

    For a single crap-app can hose T-Mobile.

    What no throttling controls built into the network?

    Or perhaps they're over-stating the issue for effect.

  7. Anonymous Coward
    Thumb Down

    Dodgy code or dodgy network?

    I'm all for blaming dodgy code in this instance, but I won't overlook how fragile T-mobile's network seems to be, if a popular - but "sucky" - app can almost bring it to it's knees.

    Of course, they won't fix it, they'll just "traffic shape" it on the cheap.

  8. Steen Hive
    WTF?

    Net Neutrality?

    Not net neutrality; load-balancing.

    What's the difference? Consider the difference between crime-prevention and racial profiling.

  9. Justin Maxwell
    FAIL

    Sledgehammer...

    ...nut

  10. Anonymous Coward
    Anonymous Coward

    Bundled data?

    Is another reason why Devs have to be cautious about making open internet connections and leaving them open.

    I am sure that there was improvements that could be made in the development but this sounds like it is down to a number of things that eventually meant that a bit of poor code could cause such a bad effect.

  11. Anonymous Coward
    Flame

    Bad practices wherever you are

    "This kind of thing is increasingly a problem as smartphones try to push battery life by shutting down data connections as quickly as possible, while the applications treat data connectivity as if it had no overhead - and thus make no attempt to aggregate traffic. Desktop applications can treat connectivity like that, resulting in a critical difference that T-Mobile reckons warrants an industry-wide exception to net neutrality."

    Hello? It's bad practice even in desktop and server applications to blast through zillions of connections, although you're more likely to see the consequences if you're doing server development (which is why various "ooh so shiny we are" desktop developers should at least familiarise themselves with the work of their server-developing counterparts).

    And an "industry-wide exception to net neutrality"? Yet another power-grab by corporations when they suddenly have to step up and do some work rather than dangle their customers upside down. You don't see T-Mobile and chums complaining when some unfortunate punter roams onto their network from abroad and starts firing up those data connections.

  12. It wasnt me
    Thumb Down

    What a load of cock.

    Dont promise users all-you-can-eat bandwidth if you can't provide it.

    And the solution isn't to create a different and artificial legal framework, that is the domain of the MPAA/RIAA etc. The solution is better development. Provide devkits that encourage/enforce efficient use of network resources.

    But don't promise (and sell) something you can't provide, then try to change the law so that you don't have to.

    1. sT0rNG b4R3 duRiD
      Thumb Up

      @It wasn't me

      "Dont promise users all-you-can-eat bandwidth if you can't provide it."

      ^ This.

  13. Anonymous Coward
    Flame

    T-Mobile FAIL?

    They said what? A demonstrably broken application was causing what amounted to a DDOS on the T-Mobile network, in circumstances which were not really normal intended operation? And T-Mobile see those circumstances as an excuse to support their case for routinely abandoning net neutrality in normal intended operation?

    Sorry T-Mobile but you can't be much more transparently wrong than that.

  14. Anonymous Coward
    Joke

    One thing to say...

    There's an app for that

  15. John Hughes
    FAIL

    That's not how IP works

    "it seems likely the application was repeatedly setting up and tearing down IP connections, using each one for a very small amount of data and thus generating more signalling load than the base stations could manage."

    1. There is no such thing as an IP connection

    2. The network shouldn't see TCP connection setup and teardown, that's for the endpoints to worry about.

    It looks like T-Mobile have a horrible layering problem. It's probably NAT.

    NAT is bad.

    This is nothing to do with network neutrality, it's to do with network stupidity.

  16. Anonymous Coward
    Pint

    @John Hughes 13:08

    You need to learn a little about the difference between connection-oriented networks (GSM, UMTS, mobiles in general) and connectionless networks (Ethernet being the classic modern example).

    I know a little, just enough to recognise that this issue has nothing to do with NAT.

    Others who know more will have to supply more details of what it *is* to do with - but it *is* to do with setting up and clearing down a connection - but closer to RF/cell level than to IP level.

  17. Robert Carnegie Silver badge

    The net neutrality angle...

    ...is that if they hadn't traced the cause of the problem and got it dealt with, their next move would probably be to identify, limit, and block the disruptive activity.

    But if net neutrality was the law, imposed on their contracts with customers, they might not be able legally to do that.

    I see their point - but as a mobile network user myself, I don't want to have my own use of online content filtered or throttled.

    1. Anonymous Coward
      Anonymous Coward

      One would assume...

      that what they are referring to here is the additional signalling in transmission network between the radio network controller and the base station. Ife the application was clearing the internet connection and thus its radio resources then immediately setting them back up, requiring additional signalling to re-establish the radio resources. The signalling channel becomes overloaded, then other requests are delay/lost then repeated leading to a cascade effect. Also T-Mobile has no control over the stack in their handsets or network its 3GPP.

      If the application is popular enough then it would become as someone said earlier a DDOS.

      Still no excuse for T-Mo bashing net neutrality.

  18. Anonymous Coward
    FAIL

    They are only upset ...

    ... because it's was lots of packets of small data. If it was lots of packets of big data we wouldn't be having this argument because 1) they'd make lots of money from overage and 2) link contention would of limited the damage of the packet storm.

  19. sT0rNG b4R3 duRiD

    Not mission critical.

    Having witnessed the gradual introduction and widespread adoption of mobile phones, through the years, I grew up with the given that mobile phones are not mission critical devices and are prone to failure, be it device failure (battery etc) or reception/network failure.

    I suppose this article is a reminder to reflect on this.

    That's the first thing.

    The second: Interestingly T-mobile is not telling us exactly what happened. One would think that if it was coding that was so blindingly bad and detrimental they would issue some sort of advisory. Maybe they have - can't find it as of now. Because of this, I would wager we cannot really comment on whether the programmer did something to be frowned upon or not.

    But I am highly suspicious that T-mobile was offering much more than it could provide and as such is blaming said programmer for making assumptions on its capability to provide.

    Clearly something needs to be done, but I am not sure that it isn't on the part of the provider.

  20. Anonymous Coward
    Badgers

    What a convenient problem for TMO!

    If this is a tower connectivity issue, then it looks like a connect/reconnect problem rather than net neutrality. Had the user stayed connected, there wouldn't be a problem, right? This looks like a radio network issue, not an IP issue. How on earth would traffic management prevent a phone from switching its radio on and off?

    How about just saying, "You may not rapidly and repeatedly connect and disconnect from our network. Otherwise will we send you a text message telling you that we will block your data connection for 2 minutes." If you want to be friendly, you could add the port numbers of the data being seen around that time, to help the user can identify the application causing the problems.

    Or you could hold/cache the radio data at the tower with the expectation that the user might connect to the same tower more than once, even if they have shut down the radio connection.

    It would be interesting to know if other network operators have the same problem.

    1. Displacement Activity

      @What a convenient problem for TMO!

      > This looks like a radio network issue, not an IP issue. How on earth would traffic

      > management prevent a phone from switching its radio on and off?

      As suggested above somewhere, the stack on the phone itself could detect this problem and deal with it, with no basestation intervention. Seems like a good idea to me.

      Base stations already have the ability to order handsets to reduce xmit power, to prevent more-local handsets from shouting down less-local handsets. It would presumably not be too difficult to use this feature to cut off offending handsets.

  21. E 2
    FAIL

    I call bullshit.

    Responding to an outbreak of a 'worm-like' app is just proper management. Such response does not require rate-tiered traffic, it requires port blocking.

  22. Gangsta
    Jobs Horns

    What!

    Sorry, What did you say?

    You want to bring down T-Mobile's network?

    Well...

    There's an App For That!

    --CEASE AND DESIST ORDER 1ZILLION --

    Dear Mr Gangsta

    The use of that phrase ("There's an App For That!") is forbidden in the perfect world; but the internet is good enough for me.

    Regards, Masterful Leader,

    Steve

  23. ykwastaken
    Jobs Halo

    Not so funny now.

    And everyone laughed when Steve said having open app dev was risky because a dev might inadvertently "bring down the west coast network".

  24. Anonymous Coward
    Joke

    Nah

    he was using broadcast ip to find the recipient....

    That'll do the trick.

  25. Anonymous Coward
    Coat

    Blofeld?

    So you're saying I can air drop a box of Android devices into city 'x' and hold them for ransom by threatening to start a script to tear down their mobile network?

    <taps fingertips together />

    Most interesting!

    (Mine's the one with the white cat in the diamond collar.)

  26. Daniel Voyce
    Go

    What was the App?

    Just wondering what this fabled app is?

  27. The Other Steve

    And it will happen again, I tell you.

    T-Mobe's shitty network aside, this is what happens if you get a load of neophyte devs with no experience other than a bit of hobbyist Java coding and let them distribute mobile apps to millions of people without adult supervision.

    This may be an acceptable price to pay for all the open, I couldn't possibly comment.

  28. Bob 18
    Thumb Down

    Use Quotas Instead

    Net Neutrality treats traffic the same no matter whether the source is Verizon, Google, etc, and no matter whether they're doing email, movies, etc.

    Quotas limit the total bandwidth given to any particular subscriber.

    A simple per-subscriber quota would solve the problem T-Mobile encountered. No need to throw out Net Neutrality and start cutting special deals with media companies, just to stop an errant messaging app.

    1. foo_bar_baz
      Megaphone

      Read above comments

      This is not about the quantity of data being transmitted. It's (most likely) about the radio network being hosed due to connections being established and torn down in rapid succession.

      1. Bob 18

        Quots on connections

        You could also have quotas on connections being established and torn down in rapid succession, not just on total bytes.

  29. dssf

    I'm not quite sure what T-Mobile is complaining about technically, but

    It seems their network is going to be downright saturated not far from now:

    http://www.androidauthority.com/index.php/2010/10/19/mytouch-4g-video-shows-off-some-features/

    If video conferencing is coming, and if simultaneous data downloads are possible, and if Android root kits allow owners to tether their phones without the disincentivizing pay-more-to-tether options, then won't TM's network see more congestion and shakeups?

This topic is closed for new posts.

Other stories you might like