Superintelligence: Paths, Dangers, Strategies - Chapter 13

Choosing the criteria for choosing

Considering that there could come a time when we could develop any arbitrary final value into a seed AI, the criteria for choosing the most important values is a vital one. Nevertheless, it is not simple to make this decision due to the countless possibilities and the uncertainty of the future due to constant change. 

The need for indirect normativity

 

In the possibility that a superintelligence attains a decisive strategic advantage, the values that we have installed to it would shape humanity. Installing proper values, then, is a necessity. 

“Indirect normativity is a way to answer the challenge presented by the fact that we may not know what we truly want, what is in our interest, or what is morally right or ideal. Instead of making a guess based on our own current understanding (which is probably deeply flawed), we would delegate some of the cognitive work required for value selection to the superintelligence. Since the superintelligence is better at cognitive work than we are, it may see past the errors and confusions that cloud our thinking.”

Coherent extrapolated volition 

Yudkowsky has suggested that a seed AI be provided the final goal of carrying out humanity’s “coherent extrapolated volition” (CEV), which he outlined as follows: “Our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”

Some explications 

Some explanations regarding the above quote include the term “Thought faster,” in Yudkowsky’s terminology, which is defined as if we were more intelligent and had thought things through more. “Grown up farther together” is defined as if we had done our learning, our cognitive enhancing, and our self-improving under conditions of adequate social interaction with one another.

Rationales for CEV

The CEV approach is supposed to be powerful and self-correcting; it is supposed to express our values rather than depending on us properly specifying every important value we have.  

“Encapsulate moral growth” 

This approach could constitute moral progress. As it has already been suggested, if programmers would provide a precise moral code for the AI to follow, humanity would leave itself with no room to change steadily to a different direction of moral convictions as society changes, which would result in no moral growth. “The CEV approach, by contrast, allows for the possibility of such growth because it has the AI try to do that which we would have wished it to do if we had developed further under favorable conditions, and it is possible that if we had thus developed our moral beliefs and sensibilities would have been purged of their current defects and limitations.”

Morality models 

CEV being a type of indirect normativity, it is not the only one. In fact, suppose that someone could create an AI with the goal of doing what is morally right, depending on the AI’s superior cognitive abilities to determine which actions are the most suitable. Bostrom asserts this suggestion as “moral rightness” (MR). Since humanity may not be able to determine moral rightness with such understanding as a superior cognitive system, we could defer that responsibility to that system. 

In the case that the AI cannot determine adequate non-relative truths about moral rightness, then it should go back to applying coherent extrapolated volition instead, or shutting itself down.

If the superintelligent system is capable of having a general ability to comprehend natural language could then be used to comprehend what we mean by “morally right.” 

Lastly, as Bostrom emphasises, it is not essential to create the ideal machine. On the contrary, we should seek to create an incredibly reliable machine, one that we can depend upon to preserve enough sanity to acknowledge its own failings. “An imperfect superintelligence, whose fundamentals are sound, would gradually repair itself; and having done so, it would exert as much beneficial optimization power on the world as if it had been perfect from the outset.”