"But the remaining very hard part is the infrastructure that goes around it. Particularly ease of use, key management, and avoiding leaking metadata. PGP-encrypted email, for example, makes no attempt to hide the source and destination, the length of the email and most implementations don't even drop all the optional clear-text headers (such as Subject)."
That was my point about it not being part of the protocols.
Take, for instance, key management and email. There's nothing in SMTP that provides for it If a hypothetical ESMTP were to replace SMTP and specified a requirement for hosting the public key (e.g. on the server pointed to by the MX record) and the mechanisms for setting and retrieving it then existing email software would be extended to provide that ease of use.
In the absence of anything to mandate the infrastructure encryption will remain an awkward add-on at best to popular email clients and mostly unused because nobody knows anyone who uses it because nobody knows anybody who uses it.