If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. The paper considers a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. Many decision-making functions are retargetable, — for every configuration under which a decision-making algorithm does not choose to seek power, there exist several reconfigurations which do —, and retargetability is sufficient to cause power-seeking tendencies. The flexibility of these results is demonstrated by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest that retargetable training procedures may train real-world agents which seek power over humans.
Analyze whether results apply to the training of other machine learning networks beyond planning- and reinforcement learning-based settings, so that the field of AI can understand the relevant safety risks before the field trains power-seeking agents.