"What's wrong with a /64 prefix?"
IPv6 expects you use bits of the prefix for subnetting - not bits of the interface identifier. Up to 16 bits of the prefix could be a subnet identifier.
You're right plain VLANs won't be an issue - I should have written subnets, not VLANs, but it's quite common to assign VLANs to subnets to isolate traffic yet being able to route traffic across subnets when needed.
Trying to use bits of the interface identifier to create subnet could create issues - not all devices may understand it when routing, and could be difficult to assign them but manually, many automatic ways to assign an IPv6 address may not let you assign some fixed bits to the interface identifier.
That's a clear difference and departure from IPv4 where you are much freer.
That's why the recommended smaller prefix assignment should be /56, not /64, even if some SOHO sites could use something smaller than a /56.