The Off-Switch Game

One of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. The goal is to study the incentives an agent has to allow itself to be switched off. A simple game between a human H and a robot R is analyzed, where H can press R's off switch but R can disable the off switch. A traditional agent takes its reward function for granted: such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. The key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility. Giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and this setting is a useful generalization of the classical AI paradigm of rational agents. These results yield three considerations for designers: an agent's incentive to accept correction stems from its uncertainty about its utility function, this uncertainty is balanced against the level of suboptimality in human decision making, and a tradeoff must be expected between the value a system can generate and the strength of its incentive to accept oversight.