Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of the objectives specified for them. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, this paper develops the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, certain environmental symmetries — where one set of choices contains a copy of another under a permutation of states — are proven sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. In these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states. Deactivating a misbehaving agent may not remain possible, because average-optimal agents would generally stop us from deactivating them if physically possible, and it is conjectured that when the discount rate is close to 1, optimal policies tend to seek power by accumulating resources to the detriment of any other agents in the environment. Real-world tasks are partially observable and learned policies are rarely optimal, so the results do not mathematically prove that hypothetical superintelligent agents will seek power, but understanding these risks is the first step in addressing them.
Address partially observable environments, suboptimal policies, or ‘almost similar’ recurrent state distribution (RSD) sets, which record how often the agent tends to visit a state in the limit of infinitely many time steps.