Superintelligence: Paths, Dangers, Strategies - Chapter 5

Decisive strategic advantage 

The question that the author seeks to answer in this Chapter is whether one particular machine intelligence system will be able to increase its intelligence so much that it obtains a decisive strategic advantage and hence, all world domination. Such a system may even use its intelligence to prevent other systems from being able to challenge it. 

Will the frontrunner get a decisive strategic advantage?

If a machine develops strong superintelligence months before another machine is able to do so, this would constitute a decisive strategic advantage and may be a sufficient enough time to tackle any other machines with potential and establish a singleton. 

According to Bostrom, it is most probable that the rise of machine intelligence will increase exponentially following the crossover point and this magnifies the possibility that “the leading project will attain a decisive strategic advantage even if the takeoff is not fast.” 

How large will the successful project be?

Subsequently, as Bostrom continued to assert, the successful project will likely be well-funded and therefore it would be a large project. “Whole brain emulation, for instance, requires many different kinds of expertise and lots of equipment.” If collective superintelligence comes to be the successful superintelligence project, it would require a great amount of networks and organizations which would thus contain much of the world economy. 

“The AI path is more difficult to assess. Perhaps it would require a very large research program; perhaps it could be done by a small group. A lone hacker scenario cannot be excluded either. Building a seed AI might require insights and algorithms developed over many decades by the scientific community around the world. But it is possible that the last critical breakthrough idea might come from a single individual or a small group that succeeds in putting everything together.”

Monitoring

The security interests behind a potential superintelligent machine are top priority, something which will incentivize governments to nationalise any project that has superintelligent potential. Another scenario is if global organizational structures are strong by the time a potential project appears, the project could be placed under international control. Consequently, as Bostrom rightly acknowledges, the question is whether international organisations and governments will identify the project with the most potential. An example of an international collaboration is the International Space Station.(ISS) 

From decisive strategic advantage to singleton 

The following are reasons that Bostrom indicates may act as a deterrent for the machine intelligence from creating a singleton. “These include non-aggregative or bounded utility functions, non-maximizing decision rules, confusion and uncertainty, coordination problems, and various costs associated with a takeover.”

Additionally, another factor is the problem of internal coordination. As Bostrom described it: “Members of a conspiracy that is in a position to seize power must worry not only about being infiltrated from the outside, but also about being overthrown by some smaller coalition of insiders.”

Lastly, costs may act as a significant deterrent. The United States had a decisive strategic advantage in nuclear weapons at the end of WWII and could have established a singleton but moral, economic, political, and human costs of initiating a nuclear war to conquer the world were too high.