"really depends on one enorrmous binary firmware"
Probably you don't know what the firmware in Linux is (and why you don't see it in Windows):
https://wiki.ubuntu.com/Kernel/Firmware
Firmware code may, and often is, not open source at all, it's strictly tied to the hardware, and the hardware manufacturer may not want to give away IP or the like. You'll need open-source CPUs, GPUs, etc...
Like it or not, any OS still needs to run atop some kind of hardware (physical or virtual), and has to program and use it as needed. Having the latest firmware loadable on-request is a plus - even CPUs can load code to fix issues.