Speculative versus parallel execution
Speculative execution is basically a way to make sequential computation faster. When the processor has to wait for, say, a condition to be decided, it makes a guess as to the outcome and starts working from that guess. If it guesses right, you save time, if not, you both lose time (for clean-up) and waste heat (for doing wasted work). You can try to work on multiple different outcomes simultaneously, but that is more complicated, and you will definitely waste work (and heat).
Speculative execution relies on very precise predictions, and these cost a lot in resources for gathering and storing statistics and analysing these. The bottom line is that speculative execution is very costly in terms of complexity and energy.
Another solution is to pause execution until the outcome is known. While this pause lasts, you can have another thread use the execution units. This is called multi-threading, and is usually implemented by having an extra (or several) copy of all registers, and schedule instructions from two (or more) threads simultaneously. You only execute instructions that are guaranteed to be needed, so there is no speculation. You can even have both threads execute instructions simultaneously, if there are no resource conflicts. The scheduling unit is somewhat more costly, as it has to look at more instructions, but it is not as bad as the complexity of speculative execution. The downside is that each thread does not run faster than if it ran alone on a processor without speculative execution, but the throughput of instructions is likely higher than this case. If the threads share cache, there is a risk of information spillage, so you generally limit this to threads from the same program.
The next step is to make multiple cores, each with their own cache. If the memory is protected (and cleared when given to a new process), this can be made safe from leakage, it scales better than multi-threading, and the complexity is lower. This is part of the reason why the trend is towards more cores rather than faster single cores. In the extreme, we have graphics processors: A large number of very simple cores that do no speculation and no out-of-order execution and which even share the same instruction stream. Sequential execution on these is horribly slow, but the throughput is fantastic, as long as you can supply a suitable workload. It is nigh impossible to make C, Java, and similar languages run fast on graphics processors, so you either need specialised languages (https://futhark-lang.org/) or call from C or Java library routines written in very low-level languages and hand-optimised.
In conclusion, the future belongs to parallel rather than speculative execution, so you should stop expecting your "dusty decks" of programs written in C, Java, Fortran, etc. to automagically run faster on the next generation of computers.
Biting the hand that feeds IT
