Intelligence and Unambitiousness Using Algorithmic Information Theory

Algorithmic Information Theory has inspired intractable constructions of general intelligence (AGI), and undiscovered tractable approximations are likely feasible. Reinforcement Learning (RL), the dominant paradigm by which an agent might learn to solve arbitrary solvable problems, gives an agent a dangerous incentive: to gain arbitrary "power" in order to intervene in the provision of their own reward. The paper reviews the arguments that generally intelligent algorithmic-information-theoretic reinforcement learners such as Hutter's (2005) AIXI would seek arbitrary power, including over us. Then, using an information-theoretic exploration schedule, and a setup inspired by causal influence theory — whose diagrams represent causal dependencies between agent and environment — a variant of AIXI is presented which learns to not seek arbitrary power; it is called "unambitious". The agent learns to accrue reward at least as well as a human mentor, while relying on that mentor with diminishing probability. And given a formal assumption — the Space Requirements Assumption, that modeling the outside-world state and the room state takes more space than just modeling the room state — that is probed empirically, eventually, the agent's world-model incorporates the following true fact: intervening in the "outside world" will have no effect on reward acquisition; hence, it has no incentive to shape the outside world. Unambitiousness is key in safely exploring the solution space to other problems that might arise, since failure modes such as deceiving the operator are not existentially dangerous and put us in a regime where we can tinker until it works. Although BoMAI (Boxed Myopic Artificial Intelligence), which maximizes reward episodically while run in a sealed room with an operator, is wildly intractable, a design for unambitiousness may be helpful once general intelligence can be approximated tractably.