If the traffic is encrypted using SSL, you only have two options available to you:
1) Look at the port number - works to some extent, but a lot of people now run their BT on port 443 (https), and if you don't want to block https you are suddenly having a very up-hill struggle between false positives and an ineffective detection rate a-la RBLs.
2) You can perform a man-in-the-middle attack, which may or may not work depending on the nature of the SSL implementation. Granted, if the server and/or the client don't have a 3rd party cryptographic certificate of their own identity, you have nothing to gauge the validity of the connection to the intended client and the MitM.
The BIG problem, however, is the CPU time required to do that much crypto in real-time. Kit that can do that in real-time for the sort of bandwidth that ISPs havev to shift is prohibitively expensive (call it £100K for a cheap cypher like RC4 for every 10Gb of traffic you have to serve, orders of magnitude more for something like AES if you also want to do something meaningful with the packet content). Also, it would become very obvious very quickly that an ISP was using it, since it would light up the warnings on your browser every time you tried to connect to legitimate SSL web sites. Either way, it hasn't happened yet, but when it does I'm sure freetards will come up with their own solution to their required webs of trust. As with other similar things, it's an arms race.