Transport for London has been forced to temporarily close its much-hyped London Underground departure times data feed due to overwhelming interest in the service. Application coders flooded the system after TfL tweaked its terms and conditions for using its datasets, allowing developers to access London's tube, bus and river …
Not all are down
'London Tube Status' on the Android still works, but it's been around for a while and maybe pre-dates the free-for-all that came about last month?
How hard can it be?
Surely the only messages the TfL feed has to display are...
Non-Summer schedule: "Sorry...if your service is even running, it'll be running late"
Summer/Tourist schedule: " We on strike until we get extra caviar with our lunches. Try walking."
Free data is great...
...but this just proves that it doesn't come for free - there's a cost to it all.
as in most cases
Hunger for data
Show's the hunger for the data, other government departments and organisations - TAKE NOTE - we want our data!
Governmental IT at work
Why is the UK government, and it's agencies, so incompetent?
They should simply acquire copies of software from Canada or Seoul, Korea (by far the better) and dispense with those less than adequate characters that populate computer rooms at present.
No data is better than slow data.
We had to destroy the user experience in order to save it.
Capacity Planning on Open Data
Its a headache, if live data is open to all without any registration etc trying to do any meaningful capacity planning is a nightmare. Simply saying the government should respond is demonstrating a lack of understanding.
How are they expected to estimate capacity
Should they just keep adding as usage goes up
Do they just cough up for capacity costs and assume its "for the good of the public"
How do they know if an app/site is to be launched with huge fanfare etc.
How do you account for badly written code that hammers the service unnecessarily
How do you cope with DOS type issues
Possibly ie suggest some form of threshold based scheme, anything over X requests per day/hour/min whatever requires registration including capacity estimate (and they get better service e.g. emails re downtime, advice on new facilities, request to add data etc). Unregistered requests over that threshold could be blocked or throttled. (that assumes they can detect it e.g. from a server hosting the mash up, not the client)
Why is every useful web service provided by the public sector always either massively undespecced? And why is every pointless web service provided by the public sectore massively overspecced?
Having run some very high load public-facing stuff..
The way to do is is to not let the public bang on your non-scaling servers. You have the vulnerable boxes behind a wall of public facing machines. The back-end machines push the data (or it's pulled from them) once per N seconds. Then the front end machines, nice fast web servers, serve the data up in a flat file stylee, and can handle millions of requests (depending on the size of the server farm).
You use load balancing on the front end servers (round robin dns is fine, nowt expensive like wanky expensive arrowpoint-style stuff needed), and you can server a hell of a lot of users without breaking the bank, and gaining fault tolerance into the bargain (if you build it properly).
End of lecture.
why not serve via someone who can scale
surely the answer is to push their non-scaling data to a cloud service - Azure or AppEngine spring to mind- and/or issue a per-app key and rate limit that key to encourage developers to cache and/or reduce their poll times to something managable, and finally tune the API so it returns as much of the data in one hit as possible vs requiring multiple hits for common functionality
Granted, I suspect the scale of the problem here is that the Tube shifts more people every hour than some countries entire rail systems so it's not a little problem.
If they are attempting to dynamically fill every request rather than using agressive caching then whoever designed this needs a pat on the head
BitTorrent. they seed one copy and drop a delta update every n-seconds. problem solved.
send my fee to .....