Superintelligence: Paths, Dangers, Strategies - Chapter 7

The superintelligent will

This Chapter wants to answer the question as to what goals will the machine superintelligence strive to achieve. 

The relation between intelligence and motivation

A ‘reductionistic’ goal such as calculating the decimal expansion of pi is simpler for humans to code and more convenient for an AI to attain, it is this type of goal that the coder will program the AI to do as efficiently as possible, without paying attention as to what the AI might do to achieve that goal. 

In this Chapter, the author does not imply that the motivations have anything to do with rationality or reason. The author is pointing out the level of intelligence of the AI which he defines as being highly-skilled at “prediction, planning, and means–ends reasoning in general.” Additionally, it is rightly recognised and we can predict with certainty that the more capable the machine is, the more it is going to be capable of attaining any relevant resources to accomplish that goal. 

Furthermore, we can also predict that “if a digital intelligence is created directly from a human template (as would be the case in a high-fidelity whole brain emulation), then the digital intelligence might inherit the motivations of the human template.” This could turn out dangerous, since the superintelligence may be hacked or corrupted and display harmful actions. . 

Instrumental convergence 

In the following headings, we will look at what the motivations of a superintelligent agent might be in accordance with its goals.

Self-preservation 

If the superintelligent agent has goals regarding the future, then it has instrumental reasons to predict the future in order to attain its future-oriented goal.  

Goal-content integrity 

It is true that it is human nature for humans to always seek survival in our final goals but on the other hand, superintelligent agents would be much more concerned about achieving their goals than surviving. This is because they can switch bodies or form exact duplicates, and therefore, for them survival is not an issue. 

Cognitive enhancement 

This is in line with the statement that was mentioned above, that the more capable the machine is, the more it is going to be capable of attaining its goals by developing the relevant cognitive skills that are required. 

Technological perfection

In this heading, it is best to keep the words of Bostrom: “An agent may often have instrumental reasons to seek better technology, which at its simplest means seeking more efficient ways of transforming some given set of inputs into valued outputs. Thus, a software agent might place an instrumental value on more efficient algorithms that enable its mental functions to run faster on given hardware.” 

Resource acquisition

Lastly, resource acquisition will very likely be an instrumental goal of the superintelligent agent, due to the fact that it would help it obtain its goals. For instance, more software and hardware resources could be utilised to operate the superintelligence faster and for a longer period of time. Additionally, more tangible resources could also be used to improve its security. “Such projects could easily consume far more than one planet’s worth of resources” and then the superintelligent agent may seek to acquire extraterrestrial resources once our technology improves and the cost of acquisition is lowered.