How many people do you need to build/design a computer?
The Cray I was built by about a dozen or so. So was the 6502. Considering you could go quite far by building something like a C-64, but with more modern production techniques, there would be a very plausible and efficient solution.
Just embrace simplicity and don't worry about efficiency at first. Efficiency is something that should only be considered early in the process if it gives you several magnitudes in speed. Split up your problems into domains, decide which domains need high security, run those on your own hardware. Find out the ones which do not and which require high speed (i.e. graphics output) and run those on isolated commodity hardware.