Bostrum, like most of his paradigm bound ivory towered ilk, completely overlooks the more fundamental aspects of the evolution of artificial intelligence.
The fact that there are now more devices connected to the Internet than people should alert us to the realization that its evolution is properly regarded as a autonomous natural process and, on the larger scale, beyond human control.
Most folk consistently overlook the reality that distributed “artificial superintelligence” has actually been under construction for many decades.
Not driven by any individual software company or team of researchers, but rather by the sum of many human requirements, whims and desires to which the current technologies react. Among the more significant motivators are such things as commerce, gaming, social interactions, education and sexual titillation. Virtually all interests are catered for and, in toto provide the impetus for the continued evolution of the Internet.
By relinquishing our usual parochial approach to this issue in favor of the overall evolutionary "big picture" provided by many fields of science. the emergence of a new predominant cognitive entity (from the Internet, rather than individual machines) is seen to be not only feasible but inevitable.
The separate issue of whether it well be malignant, neutral or benign towards we snoutless apes is less certain, and this particular aspect I have explored elsewhere.
Stephen Hawking, for instance, is reported to have remarked "Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all,"
This statement reflects the narrow-minded approach that is so common-place among those who make public comment on this issue. In reality, as much as it may offend our human conceits, the march of technology and its latest spearhead, the Internet is, and always has been, an autonomous process over which we have very little real control.
Seemingly unrelated disciplines such as geology, biology and "big history" actually have much to tell us about the machinery of nature (of which technology is necessarily a part) and the kind of outcome that is to be expected from the evolution of the Internet.
This much broader "systems analysis" approach, freed from the anthropocentric notions usually promoted by the cult of the "Singularity", provides a more objective vision that is consistent with the pattern of autonomous evolution of technology that is so evident today.
Very real evidence indicates the rather imminent implementation of the next, (non-biological) phase of the on-going evolutionary “life” process from what we at present call the Internet. It is effectively evolving by a process of self-assembly.
The "Internet of Things" is proceeding apace and pervading all aspects of our lives. We are increasingly, in a sense, “enslaved” by our PCs, mobile phones, their apps and many other trappings of the increasingly cloudy net.
We are already largely dependent upon it for our commerce and industry and there is no turning back. What we perceive as a tool is well on its way to becoming an agent.
There are at present an estimated 2 Billion Internet users. There are an estimated 13 Billion neurons in the human brain. On this basis for approximation the Internet is even now only one order of magnitude below the human brain and its growth is exponential.
That is a simplification, of course. For example: Not all users have their own computer. So perhaps we could reduce that, say, tenfold. The number of switching units, transistors, if you wish, contained by all the computers connecting to the Internet and which are more analogous to individual neurons is many orders of magnitude greater than 2 Billion. Then again, this is compensated for to some extent by the fact that neurons do not appear to be binary switching devices but instead can adopt multiple states.
Without even crunching the numbers, we see that we must take seriously the possibility that even the present Internet may well be comparable to a human brain in processing power.
And, of course, the degree of interconnection and cross-linking of networks within networks is also growing rapidly.
The emergence of a new and predominant cognitive entity that is a logical consequence of the evolutionary continuum that can be traced back at least as far as the formation of the chemical elements in stars.
This is the main theme of my latest book "The Intricacy Generator: Pushing Chemistry and Geometry Uphill" which is now available as a 336 page illustrated paperback from Amazon, etc .