"Can someone remind us again why the Internet Engineering Task Force decided not to make this next-gen networking protocol backward-compatible?"
Who knows. IPv4 was, kinda, a bit of a kludge even for its time, with limitations built in that are akin to "nobody will ever need more than 640k"...
Perpetuating that is undesirable. However, I'm not convinced that the replacement is kludge free either.