Superintelligence cannot be contained: Lessons from computability theory

Superintelligence is a hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds. In light of recent advances in machine intelligence, a number of scientists, philosophers and technologists have revived the discussion about the potential catastrophic risks entailed by such an entity. This article traces the origins and development of the neo-fear of superintelligence, and some of the major proposals for its containment. Such containment is, in principle, impossible, due to fundamental limits inherent to computing itself. Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world, strict containment requires simulations of such a program, something theoretically (and practically) infeasible. Formal verification techniques, although fruitful for restricted classes of programs and inputs in safety-critical applications, “cannot be useful to the containment of superintelligence,” because superintelligent machines are written in Turing-complete programming languages, are equipped with powerful sensors, and take the state of the world as input, rendering those techniques “unusable at this grand scale.” Via Rice’s theorem, “we may not even know when superintelligent machines have arrived,” since deciding whether a machine displays superintelligence is undecidable; and computability theory reveals “fundamental, mathematical limits to our ability to use one AI to guarantee a null catastrophic risk of another AI.”