Flagged as work that qualifies or bounds the uncontainability case.
Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. This paper studies when an RL agent has an instrumental goal to tamper with its reward process, and describes design principles that prevent instrumental goals for two different types of reward tampering: reward function tampering (inappropriately influencing the implemented reward function) and RF-input tampering (inappropriately influencing the information that the reward function has about the environment state). These design principles include current reward function optimization (optimizing rewards assigned by the current implemented reward function), uninfluenceable learning (the expected output of the reward-function learning process is the same regardless of the agent's actions), history-based rewards (reward functions with access to the full history of actions and observations), and belief-based rewards (rewards based on the agent's belief about the underlying state). Combined, they can prevent both types of reward tampering from being instrumental goals. The analysis benefits from causal influence diagrams — a graphical technique for analyzing agent incentives that combines causal graphs and influence diagrams — to provide intuitive yet precise formalizations. Isolating or encrypting the reward process will not scale indefinitely with agent capabilities, as a sufficiently capable agent may find ways around most defenses; instead, an important next step is to turn the design principles into practical and scalable RL algorithms, verify empirically that they do the right thing in setups where reward tampering is possible, and let them evolve into a set of best practices for how to design capable RL agents. The principles leave open the problem of specifying a good reward function in the first place.
Turn the design principles into practical and scalable reinforcement learning (RL) algorithms, and verify empirically that they do the right thing in setups where reward tampering is possible. Evolve these design principles into a set of best practices for how to design capable RL agents.