Comments on EAK's and prone's comments
Mr. Horizontal (prone!): HTML was developed by a physicist, not a computer scientist, to help communicate at CERN with a simple mark-up language authored by hand using a text editor. While many terse and efficient, binary, document+graphics page mark-up languages already existed, most were proprietary (Interleaf, Frame, MS Word, PageMaker, Quark, WordPerfect etc..), Bernars-Lee chose to create his own, modeled from the most verbose and inefficient mark-up language, SGML; a product of committee meant to satisfy every need and an ISO standard.
HTTP is used outside its scope because it was the easiest way to get data through most firewalls. Additionally, it survives "traffic shaping" and QoS better than any UDP traffic on non-well known ports, but not as well as VoIP.
You are 70% wrong about HTTP not maintaining a connection. Only embedded use the simpler V1 implementations. V2 allows connection persistence. Pipelined requests (doing GETs on new objects before outstanding ones have been sent) happen far less than they should due to all the layers and components of software constructing and sending content from servers - its just less bug-prone and complicated for each bloated package to serve its part on its own connection. HTTP connections are also kept open using a technique called browser comforting which tells browsers to not time out the GET, more data is on the way.
Google's changes also accommodate (encourage?) that inefficient and obese server software that's too complicated to funnel disparate pieces over already established connections. Google's changes supply another out for bad software, besides all those revenue generating land mines on pages.
Eric, you made some interesting points. Virtual servers can be a blessing. With only 65K TCP port numbers(16 bits in IPV4) for an IP address and being required to wait before reusing closed ones, running out sometimes is a reality previously solved by load balancers to proxy connections to multi-hosted server interfaces (a pool of IP addresses to use).
You are spot on about interaction issues. TCP window sizes are listed in the Google paper for various implementations. Vista and Win7 are tiny compared to the others, especially MacOS. There are actually two reasons for it. The built-in QoS scheduler is the main one. Higher priority packets will have a long queuing delay to get serviced if it has to wait a long time for a gap in a long HTTP stream. It has more to do with behavior of the server and devices in between than the client. Microsoft plays it safe such that the high priority applications get good service despite unknown and uncontrollable devices. The workaround for poor network file sharing speeds on LANs (where "feature" noticed the most) while listening to streaming audio was turning off streaming multimedia priority in the registry.
I've seen a bug in another Unix stack that caused a problem like the one you have with IRIX. Using Wireshark, I saw that the server had sent all the data followed by a FIN, saying it had no more data and was ready to close. The client didn't get one of the last packets, and ACKed only up to before the problem. The server responded with the missing packets OUT OF ORDER, followed by a FIN. Repeat infinitely, connection appears "hung". If the client was able to re-order packets, this would not be a problem. I think it was a Linux client, so an OS used as a server can't spare the time and memory to hold some number of packets to sort, hopeful that it has missing pieces. If either the bug didn't exist or the client did reordering, all would have been fine.