Unexplainability and Incomprehensibility of Artificial Intelligence

Explainability and comprehensibility of AI are important requirements for intelligent systems deployed in real-world domains both because users want and frequently need to understand how decisions impacting them are made and for safety and security reasons. In this paper, two complementary impossibility results are described: Unexplainability and Incomprehensibility. Respectively, advanced AIs would not be able to accurately explain some of their decisions and, for the decisions they could explain, people would not understand some of those explanations. These results present a number of problems for AI safety: evaluation and debugging of intelligent systems become much harder when their decisions are unexplainable or incomprehensible, and, in case of AI failures, accurate explanations are necessary to understand the problem and reduce the likelihood of future accidents. If we grow accustomed to accepting AI’s answers without an explanation, essentially treating it as an Oracle system, we would not be able to tell if it begins providing wrong or manipulative answers.