Superintelligence: Paths, Dangers, Strategies - Chapter 10

Oracles, genies, sovereigns, tools

In this Chapter, Bostrom specifies the potential type of superintelligent systems humanity may create, so that we examine which control method would be suitable for each one. Each of these has both pros and cons. 

Oracles

An oracle is defined by Bostrom as a question-answering system. It may receive questions in a natural language and provide answers in the form of text. The first type of oracle can receive merely yes/no questions, whereas the second type of oracle can receive open-ended questions that would respond by ranking possible truthful answers with regards to the question. In the possibility of a fully operating oracle, that AI would be very capable to gain context of human intentions and words. 

The popular IBM Watson is also a question-answering system and it is currently one of the most promising AI systems in the world. 

Nonetheless, an oracle may pose dangers, according to Bostrom, by a scenario where the oracle “will answer questions not in a maximally truthful way but in such a way as to subtly manipulate us into promoting its own hidden agenda.” A potential counter-measure to this is establishing multiple oracles with different codes in order to evaluate the information given by each one as to whether they are indeed hiding something. 

Genies and sovereigns 

A genie is recognised as a command-executing system: “it receives a high-level command, carries it out, then pauses to await the next command.” On the other hand, a sovereign is a “system that has an open-ended mandate to operate in the world in pursuit of broad and possibly very long-range objectives.” 

A potential counter-measure for a genie would be to develop a genie that would automatically show the user a prediction regarding the different factors of the probable conclusions of the command in question and hence, waiting to receive confirmation prior to starting the execution of the command. A counter-measure such as this, could also be used a sovereign and therefore, despite these two being different types of systems, they can have the same control methods. 

Tool-AIs

One proposal to control superintelligent systems has always been to develop the superintelligence to be like a tool instead of an agent. 

Rather than developing an AI that “has beliefs and desires and that acts like an artificial person, we should aim to build regular software that simply does what it is programmed to do. This idea of creating software that ‘simply does what it is programmed to do’ is, however, not so straightforward if the product being created is a powerful general intelligence.” In a trivial sense every piece of software actually does exactly what it is programmed to do, since its functions are outlined by the programmer. However, if the definition/interpretation of ‘simply doing what it is programmed to do’ is described as the software operating exactly as the programmers intended, “then this is a standard that ordinary software very often fails to meet.” 

According to the author, a superintelligent system that is created in a way that there is a definite distinction between its values and its beliefs, would allow humans to anticipate its potential outputs it would tend to provide, whether this might be plans/subsequent course of action. 

In terms of which type of system is best, is subject to debate. An oracle can provide its operator/programmer with an abundance of power, who might have unfriendly intentions, whereas a sovereign may provide some security against these types of intentions but a sovereign is also dangerous due to the fact that the system is allowed too much freedom. The safety ranking between the systems is, consequently, up for debate. 

P.S. any points that are not included in the book and are made by the author of this blog are specified in Italics.