Perhaps I've mis-understood or mis-represented myself.
I wasn't thinking about (and I'm not concerned about) anything that involves inspecting or re-writing the packet header. I don't think there is anything private there and my understanding of the specs is that the header is, by design, the bit that intermediate parties are expected to read and modify as part of the delivery process. The packet body, however, is the opposite in just about every way. Nothing beyond "blind copying" should be necessary and therefore anything beyond "blind copying" is evil.
For an ISP to use source or destination addresses, or IP protocol numbers, or TCP/UDP port numbers, as part of its traffic shaping strategy is probably both defensible and open to challenge.
Gratuitously dropping TCP packets, for example, *will* just result in requests for re-transmission, whereas dropped UDP packets are generally just lost. It seems reasonable for an ISP to consider that when deciding what to drop. Similarly, if the port number suggests a higher level protocol that can wait (like FTP) it would be fair to let that packet wait.
Conversely, filtering on the IP address to favour your own services would be like an OS vendor favouring the products from their own applications division. In this case, the ISP would be leveraging their monopoly position in "shunting data around" (at least, it's a monopoly with regard to this particular customer) to extend that dominance into a new market, such as "IP telephony".