Discover theknowable limits.

“Artificial General Intelligence (AGI) undergoes evolutionary pressures to expand its material base, which converges over the long term on making the Earth uninhabitable for biological life. It is impossible in principle to build AGI such that it is compatible with biological life.”

“It is impossible to precisely and consistently predict what specific actions an intelligent system smarter than humans will take to achieve its objectives, even if its terminal goals are known, because predicting all decisions of a superintelligence would require equal intelligence.”

“Advanced AIs cannot accurately explain some of their decisions, and, for the decisions they can explain, humans would not understand some of those explanations.”

“AGI safety is impossible in principle: all alignment methods reduce to error detection and correction, and the minimum extent of correction necessary to correct for all AGI effects relevant to long term human survival exceeds the extent possible by any available method.”

“Complete control, defined as full predictability of AI actions and total adherence to safety requirements, is unattainable. Instead of pursuing complete AI containment, resources should be allocated to risk management strategies that acknowledge AI's unpredictability and prioritize adaptive oversight mechanisms.”

“Impossibility theorems applicable to AI can be assigned to five categories based on mechanism: deduction, indistinguishability, induction, tradeoffs, and intractability. Deductive impossibilities deny 100% guarantees for security.”

“In most cases traditional agents have an incentive to disable their off switch, however a robot that is uncertain about the human's utility function, and treats the human's decision as evidence about it, has a positive incentive to allow itself to be switched off, provided the human is sufficiently rational.”

“Safe development of artificial general intelligence requires testing, but testing itself may be unsafe because some natural goal systems create emergent incentives to tamper with the test environment, make copies of themselves, or manipulate developers.”

“Strict containment of a superintelligence is, in principle, impossible, due to fundamental limits inherent to computing. No algorithm can determine whether an arbitrary program will cause harm.”

“Most reward functions make it optimal to keep options available and, when maximizing average reward, to navigate toward larger sets of potential terminal states.”