Why did they have to completely redesign the protocol for IPv6? All that we needed was a few more bits for the address fields
Well, for a starter, adding a few extra bits (bytes) to the address field will break things. If you're going to break things, it's better to engineer something new and completely incompatible rather than just using a sticking plaster approach.
before opening a huge security hole into a home or office
NAT is not security, especially given that uPNP NAT traversal exists. Also, you're not going to run IPv6 without firewall rules on your router, are you? And anyway, I'd wager that most security problems don't come from attacks from outside the network but from numpties inside it.
Instead with IPv6 we have wacky huge address fields
Which, if (as you should) you assign addresses randomly, improves your protection against network mapping and hence port scanning, even if you do accidentally forget to do ingress filtering. You can't scan what you can't find, and /64 is plenty big of a space to hide a few hundred machines in.
massive /64 allocations to small businesses as standard
A /64 allocation is for a LAN. Actual allocations to businesses would be larger, like a /48, so you can have up to 65535 LANs.
Why did they have to completely redesign the protocol for IPv6?
Coming back to the first point, although I'm not sure about how well this works right now, IPv6 also has features for device mobility. So new protocols open up the possibility of new ways of working.