Limitations on Accurate, Trusted, Human-level Reasoning

The paper identifies a fundamental incompatibility between the goals of accuracy, trust, and human-level reasoning in artificial intelligence (AI) systems, for strict mathematical definitions of these notions. Accuracy of a system is defined as the property that it never makes any false claims when it has the ability to abstain from making a prediction on any input, and trust as the assumption that the system is accurate. Human-level reasoning is defined as the property of an AI system always matching or exceeding human capability. The core finding is that - for the formal definitions of these notions - an accurate and trusted AI system cannot be a human-level reasoning system: for such an accurate, trusted system there are task instances which are easily and provably solvable by a human but not by the system. Key to the proof is the formalization of the notion of trust, which allows separating the intrinsic property of a system (being accurate) from its epistemic status (being trusted). Attempting to circumvent the result by augmenting or retraining an AI system “only pushes the problem one step further,” since the self-referential construction can be repeated against any successor system.