back to article FCC taps CenturyLink on shoulder, mumbles about a fine for THAT six-hour 911 outage

US comms regulator the FCC has fined CenturyLink $16m (£10.8m) for a network outage that left 11 million Americans unable to make 911 calls. CenturyLink made a $772m (£520m) profit in 2014. Last month, the watchdog charged Verizon $3.4m for its part in the downtime – and this week has smacked down the ISP and telco CenturyLink …

  1. Anonymous Coward
    Anonymous Coward

    Should count themselves luck it was only 6 hours, normally whenever we try to get something done by them it's delivered a month late and doesn't work.

  2. Martin Gregorie

    What is this "come down hard" of which you speak?

    A $10M fine on a profit of $770M is just over 1%.

    Thats just a brush with a feather duster that they'll never notice. If the FCC was serious they'd bong them 25% of last year's profit.

  3. James O'Shea

    They didn't notice for _six hours_?!

    "The problems started on April 9, last year, just before midnight at a 911 call routing station in Englewood, Colorado, which was responsible for transferring emergency calls to eighty-one 911 call centers in California, Florida, Minnesota, North Carolina, Pennsylvania, South Carolina, and Washington.

    There was a backup system for the 911 calls located in Miami, but the outage failed to set off an operator alarm, and so the switchover didn't happen. It was another six hours before the fault was noticed and the switch made, at which point normal service was restored."

    The boyz.n.grrlz on 911 duty in places like Miami and Pittsburg and Charleston and Seattle DIDN'T NOTICE THAT THEY WEREN'T GETTING ANY CALLS FOR SIX HOURS? Or, worse, they did notice, and nothing was done? This doesn't call for a fine, this calls for prison time. And seeing as it's interstate, this means _Federal_ prison time. Lots of it. What the _hell_ happened to the failover system? Was it ever tested? Bigger fine, and more prison time.

  4. Colin Miller

    I'd agree with James. There should be automatic systems at the 911 call centre (the bit that asks which emergency it is), that monitors the geographic rate of calls. If an area hass far lower than would be expected for the time-of-day and day-of-week, then an alert is sent, so that someone can look at it.

    Likewise, at the subscriber-most exchanges, test calls should be made every, say 15 mins, to make sure that the call connects to an operator. If the call fails to connect, an alert is raised.

    If the alerts aren't acknowledged, then it should be sent to another response centre, so they can prod the correct centre to fix it.

  5. Alistair
    Boffin

    errrrrrr

    I don't think that PSAPs operate *quite* the way the above commentards think.

    If you go read the actual report -

    "a total of 6,410 calls to 911 attempted, of which 5,618 failed in the Washington, North Carolina, and Minnesota, as described in the table below. Approximately one thousand additional calls failed in California, Florida, Pennsylvania, and South Carolina"

    -- so - yes there were some serious drops, but I'll guarantee that the volume at the destination did not drop off as far as you think. Furthermore the target list is fairly short as providers go - so - other than Washington State PSAPs (who apparently DID call appropriate folks) most of them were still busy.

    I know from our e911 system - that cdr (the actual event record) gets handled separately, and if the rest of the sequence isn't there (the connection completing and so on) alarms go off. And loud alarms. When last I was looking at that bit, we had 8 destination centres for those connections.

    There is a *crapton* of failure here, but it is most definitely NOT with the PSAP staff. They're too damn busy answering those calls. ( A good friend ended up on PTSD leave after 5 years in one of those hells. You really do need a special sort of insane to do that job)

    The kicker here is that the cause was a DB unique key limit. Fuck.

  6. Crazy Operations Guy

    "a 911 call routing station"

    The fuck... I'd always thought that the *local* switch was wired directly to the closest 911 response center and a regional link would only be used when that call center was down (Even then it should be wired to connect to a secondary center in the area). But I suppose its my fault for assuming that CenturyLink would be logical in their operations...

    If I were a customer of theirs, it would mean that my call was traveling 1100 miles to get to the routing station only for it to travel another 1100 miles to the 911 center that's less than a mile from my house. I might as well just call the police / fire station directly...

  7. Mark 85
    Meh

    CenturyLink... formerly known as Qwest... Had them as both names, they sucked.

  8. Tom 13

    So

    CenturyLink - $16m

    Verizon - $3.4m

    Intrado Communications - $1.7m fine

    But, having read the actual report on the earlier article, El Reg is still burying the real lead here. The FAULT was at Intrado Communications. Whatever their faults, both Verizon and Century had subcontracted the 911 work to Intrado who sold themselves as the experts in handling 911 call routing. The people most responsible for the entire fuster cluck have paid the least amount in actual damages.

    1. James 100

      Re: So

      One way or another I suspect Verizon and CenturyLink will be wanting to claw those fines back from Intrado now, since this resulted from their screwup, either in the form of "you just cost us $16m, hand it over" or next contract renewal going along the lines of "last time we paid you $x, but you screwed it up and cost us $16m, so we'll knock that off the fee this time OK?"

      Whatever happens with the fines, I just hope they implement better monitoring to detect outages like this in future.

      (Funny, I remember being assured by smug UK regulator-drones a few years ago that 911 just doesn't work reliably in the US because there are too many local calls clogging the exchanges. I also remember the reaction when I mentioned that claim to someone at a US telco at the time...)

  9. Henry Wertz 1 Gold badge

    And the irony...

    And the irony of it is, since nobody has any use for a landline any more, CenturyLink's main method of trying to sell landlines (other than force-bundling them with services people may still have a use for like DSL), is implying that a landline is much better for making 911 calls than making them from a cell phone or VOIP service.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like