Utility functions or their equivalents (value functions, objective functions, loss functions, reward functions, preference orderings) are a central tool in most current machine learning systems. These mechanisms for defining goals and guiding optimization run into practical and conceptual difficulty when there are independent, multi-dimensional objectives that need to be pursued simultaneously and cannot be reduced to each other. Ethicists have proved several impossibility theorems that stem from this origin; those results appear to show that there is no way of formally specifying what it means for an outcome to be good for a population without violating strong human ethical intuitions (in such cases, the objective function is a social welfare function). This is a practical problem for any machine learning system (such as medical decision support systems or autonomous weapons) or rigidly rule-based bureaucracy that will make high stakes decisions about human lives: such systems should not use objective functions in the strict mathematical sense. The paper explores the alternative of using uncertain objectives, represented for instance as partially ordered preferences, or as probability distributions over total orders. Previously known impossibility theorems can be transformed into uncertainty theorems in both of those settings, and the paper proves lower bounds on how much uncertainty is implied by the impossibility results. The paper closes by proposing two conjectures on the relationship between moral certainty and instrumentally convergent subgoals: the Totalitarian Convergence Conjecture, that powerful agents with mathematically certain, monotonically increasing, open-ended objective functions will adopt sub-goals to disable or disempower other agents in all or almost all cases; and its converse, the Pluralistic Non-Convergence Conjecture, that powerful agents with mathematically uncertain objectives will not adopt sub-goals to disable or disempower other agents unless those agents constitute a probable threat to a wide range of objectives. Machine learning researchers should avoid totally ordered objective functions or loss functions as optimization goals in high-stakes applications because systems designed that way suffer from an ethical “type error” and high-stakes systems should always exhibit uncertainty about the best action in some cases. The emergence of instrumental convergence is deeply connected to moral certainty: agents that are not completely sure of the right thing to do are much more likely to tolerate the agency of others, a pattern also observed in human ideologies, where totalitarianism has often been built on a substructure of purported moral certainty.
Investigate other formalizations of ethical uncertainty theorems for objectives, beyond partial orders over states or probability distributions over total orderings.