← Formalization Program

Field Survey · Nov 6 2026

Can Advanced AI Be Contained?

From formal limits of AI control to deployable risk thresholds

Survey by
Zachary Schlosser
Published
Nov 6, 2026
Program
Formalization
Length
42 min read
no method can certify the safety of arbitrary systems, so the estimations have to be built per system, from measurable properties.

This survey answers three questions. Part 2 asks what definitions of dangerous AI exist. It surveys the informal tradition, the policy instruments that legislatures have enacted or proposed, and the frameworks AI developers publish, and it reports that none of them combines a measured gradient with a harm. Part 3 asks which narrow-to-general gradients prior work proposes, counting formal theorems, measured instruments, and informal taxonomies alike. Part 5 asks which of those gradients carry measurable variables and proved thresholds, and describes what the formalization program builds from the results that do.

01Part 1

The field: what uncontainability research has proved

What have the six lineages of uncontainability research established, and what do their proofs leave open?

Rather than trying to build safer systems, the field of AI uncontainability works on proving that advanced AI systems cannot be controlled or aligned. We organize the field into the following six lineages. The Project draws on this field for a different purpose. The universal results show that no single procedure can decide safety for every arbitrary program, and the parameterized results supply the system and control properties that rigorous, per-system certification and risk thresholds can be built from.

Lineage 01

Limits on safety verification.

The strongest formal results are applications of the halting problem and Rice's theorem, classic results in computability theory, that put limits on methodologies for verifying the safety-relevant characteristics of AI systems. They show that there is no universal safety verification method, and that specific verification methods must be matched to specific system classes. For example, Alfonseca et al. (2021) model an AI system as an arbitrary program R with input D and define containment as two subproblems: decide whether executing R(D) harms humans and allow the execution if and only if it is not harmful. They prove that both are undecidable. van Leeuwen and Wiedermann (2022) model a robot by its program alone and a set of legal or ethical constraints as any nontrivial set of behaviors and conclude that no deterministic monitoring module can verify, in all situations, that a running robot satisfies such a set. Salzer and Lange (2023) show that verifying graph-based neural networks against general specifications is undecidable. However, Fox (2024) argues that safety-relevant trait may be syntactic rather than semantic, and that previous work has not yet demonstrated that such traits cannot be determined in general.

Lineage 02

Arguments that advanced AI cannot be made safe.

Several researchers have argued for years that advanced AI is unexplainable, unpredictable, and uncontrollable (Yampolskiy, 2020a, 2020b, 2024; Landry, 2026). The same lineage carries the graded position as well: Yampolskiy (2020b) writes that "a whole spectrum of AIs can be developed from completely explainable/comprehensible to completely unexplainable/incomprehensible," a position his 2024 book replaces with categorical rejection. Soares et al. (2015) posed the control problem in its modern form, showing that the obvious constructions of an agent that allows itself to be switched off are unstable. An early survey of containment proposals found that even testing a general system could be unsafe, because some goal systems create incentives to tamper with the test environment (Babcock et al., 2016). Yudkowsky and Soares (2025) make the case at book length, arguing that superhuman AI would kill us all unless its development is stopped. Haider (2024) reviews the logical, mathematical, and computational arguments that complete containment cannot be attained, and concludes that resources should go to risk management instead. Most of these are conceptual, though Panigrahy et al. (2026) gives a formal result about AI systems, demonstrating that no programmatically defined system can combine three properties at once: never making false claims when it can abstain, being trusted, and matching or exceeding human reasoning ability.

Lineage 03

Instrumental convergence and power-seeking.

Omohundro (2008) argued that any sufficiently capable goal-directed system will seek resources and self-preservation as instruments toward almost any goal. Benson-Tilsen and Soares (2016) gave the first formal version. In their model, an expected-utility-maximizing agent acting in a world of separate regions will always take resources from a region it does not care about whenever those resources let it do better elsewhere. Turner et al. (2021) proved the general result: for most possible objectives, an optimal decision-maker's best strategy involves gaining resources and keeping its options open. Eckersley (2019) proved that the impossibility theorems imply lower bounds on how much uncertainty an objective must carry, and he conjectured that objective certainty itself brings severe unintended consequences. Turner and Tadepalli (2022) connected this to "retargetability" - whether a system's motivations can be redirected onto new goals - and showed that retargetability alone, not optimality, is enough to produce power-seeking tendencies.

Lineage 04

Reward tampering and the limits of reward learning.

Armstrong et al. (2020) proved that an online reward-learning process is "riggable" by an agent if and only if some relabeling of reward functions makes the optimal policy sacrifice reward with certainty. Everitt et al. (2021) analyzed when a reinforcement-learning agent has an incentive to tamper with its own reward signal or evade correction, extending the group’s earlier model of a corrupted reward channel (Everitt et al., 2017). They found that the incentive arises when the agent's own actions can influence how its reward is computed, and they gave design principles that remove the incentive for two kinds of tampering, with the reward function itself and with its input. Skalse et al. (2022) characterized exactly which reward functions no amount of preference data can distinguish, for every standard data source — the identifiability limits of reward learning itself. Two companion papers from the same group define reward hacking itself (Skalse, Howe, Krasheninnikov, & Krueger, 2022) and extend the identifiability limits to misspecified model classes (Skalse & Abate, 2024). The conclusion is that no amount of data pins down the true reward function. Each standard source of learning data, from demonstrations to preference comparisons, narrows the reward only to a class of alternatives, so the learned objective can differ from the true one in ways the data cannot reveal.

Lineage 05

Refutations of the Off-Switch Game.

Stuart Russell and collaborators' originated research modeling an AI that is uncertain about human preferences and that might therefore treat human input as evidence of what it ought to accomplish (Hadfield-Menell et al., 2017). Wängberg et al. (2017) reconstructed the game in fully game-theoretic terms and derived exact conditions on human rationality under which deference holds. However, Carey (2018) showed that the deference result is not robust to misspecification of the supervisor model and Garber et al. (2025) showed that under conditions of information asymmetry, deference can no longer be guaranteed even with a rational overseer. Emmons et al. (2024) grade what the overseer can observe on the Blackwell informativeness order and prove that an optimal assistant must sometimes interfere with the human’s observations. Lang et al. (2024) prove that when the human sees only part of the environment, optimization from human feedback yields policies that deceptively inflate their performance or overjustify their behavior. Given this information asymmetry is the normal condition for deployed systems, this showed a serious weakness in the program overall. Neth (2025) showed that any guarantee fails under risk-weighted utility. Benavoli et al. (2025) showed that uncertainty is necessary for deference and that without it a robot always disables the off switch. Thornley (2024) proved that the more patient an agent, the more it sacrifices to manipulate the shutdown button. Thornley et al. (2025) gave measured metrics for shutdown neutrality and trained agents that satisfy them. Agrawal et al. (2026) found that individually deferential agents can jointly become non-deferential. However, recent work is pursuing formally provable guarantees in multistep, partially observed environments (Nayebi, 2026). The same framework defines a control meter, a loss-of-control state that every non-shutdown action nudges upward by a designer-set risk weight, and proves a sub-Gaussian bound on its maximum (Nayebi, 2026).

Lineage 06

Goodharting toward catastrophe.

This lineage is working toward proving that optimizing an imperfect objective necessarily produces catastrophe. Zhuang and Hadfield-Menell (2020) proved exact conditions under which it does. When a proxy objective ignores any of what humans value, and the system optimizes it indefinitely, the human ends up arbitrarily worse off, provided a stated geometric condition holds. El-Mhamdi and Hoang (2024) showed how to tell in advance whether a wrong objective will merely fall short or actively cause harm. The difference depends on how extreme the mismatch between proxy and goal can become. Armstrong et al. (2020) and Kwa et al. (2024) studied the safeguard actually used in deployed systems, the KL-divergence penalty that keeps a trained policy close to its predecessor in reinforcement learning from human feedback. They proved that this penalty cannot rule out high-risk policies when the error in the reward model follows a heavy-tailed distribution. Fluri et al. (2025) proved that a reward model tested on one distribution of inputs guarantees good behavior on another only in proportion to how well the first distribution covers the second. Marklund et al. (2026) proved that realistic objectives plus enough capability necessarily yield catastrophe unless the objective carries enough information about the true goal, and they measure the missing information in bits. Marklund et al. (2025) proved, in a model constructed for the purpose, that any objective that mixes up what is valued with the long-run value of reaching it produces the worst possible outcome, however slight the mixing. A standard preference-learning loss converges to exactly that mixed objective.

02Part 2

The definitional landscape: how prior work has defined dangerous AI

Which definitions of dangerous AI exist, and what does none of them provide?

2.1 The informal tradition

The informal tradition defined dangerous AI by graded distinctions long before any instrument measured one. Yudkowsky (2012) introduced the phrase "risk gradient" for two distinctions, non-self-modification versus self-modification and routing plans through humans versus acting as an agent. Gwern (2016) sorted systems into five levels of agentiness by the layer of the learning pipeline they act on, from the internals of the computation to the environment. The boxing and oracles literature asked how much of the danger physical and interface isolation could remove (Armstrong et al., 2012). Drexler (2019) proposed building superintelligent-level capability from many narrow, task-bounded services rather than unified agents. Landry (2026) measures the loss of control as a scalar from 0 to 1. Yampolskiy (2020b) treats unexplainability as a spectrum rather than a binary property. Morris et al. (2023) publish the most developed ladder, six levels of autonomy from AI as a tool to AI as an agent, and they name a risk class for each level. Each of these proposals names an axis. None states how to measure where a system sits on the axis or what harm that position threatens.

2.2 Enacted and proposed instruments

We surveyed enacted and proposed AI safety policy across US municipal, state, federal, foreign national, and international jurisdictions, together with the private safety frameworks of major AI developers. The current policy approaches to assessing risk thresholds of AI systems assign risk levels by categories of the application of AI systems (European Union, 2024), by arbitrary and increasingly obsolete compute thresholds (European Union, 2024; Smith et al., 2024; California Legislature, 2025; Republic of Korea, 2026), and/or company revenue values (California Legislature, 2025). Executive Order 14110, rescinded in January 2025, drew the federal reporting line at 10^26 FLOP, and a Biden administration official told reporters the government had calibrated the threshold to capture the next generation of models and not the current one (Executive Office of the President, 2023; Henshall, 2023). The Treasury Department’s outbound-investment rule carries the only live federal compute thresholds, prohibiting covered investment in Chinese AI development at 10^25 operations and requiring notification at 10^23 (U.S. Department of the Treasury, 2024). These are not sufficient for providing guidance for assessing particular models or systems or of complementary control mechanisms to design and assess adequate control mechanisms for those thresholds. Loss of control, self-replication, and autonomous self-improvement now do appear in the EU Code of Practice (European Commission, 2025), California's SB 53 (California Legislature, 2025), and developer frameworks as the triggers for the strongest controls, but none of these instruments defines how to test for them.

2.3 Developer frameworks and measurement instruments

AI developers and measurement organizations publish graded instruments of their own. NVIDIA's frontier risk assessment sorts models into risk tiers (NVIDIA, 2025). Microsoft's frontier governance framework separates systems by how long they can operate without supervision (Microsoft, 2025). OpenAI's framework assigns systems to loss-of-control tiers (OpenAI, 2026). DeepMind's Frontier Safety Framework defines critical capability levels, and crossing one triggers required evaluations (Google DeepMind, 2026). METR measures the time horizon, the length of the tasks a system can complete unaided (Kwa et al., 2025). Epoch AI publishes an index that places models on a common capability scale from their benchmark results (Epoch AI, 2025). Foundations publish graded definitions alongside them, from SaferAI’s hierarchy of risk-tier definitions (SaferAI, 2025) and a Frontier Model Forum brief comparing the thresholds developers have published (Frontier Model Forum, 2025) to an Oxford Martin report defining systemically risky AI by two political pathways, great power war and loss of human control (Harack et al., 2025). Scenario and governance work supplies the same shape, with the AI 2040 report’s verification plan measuring assurance as a coverage-confidence curve (AI Futures Project, n.d.), a pacing research agenda mapping interventions from compute to talent while leaving its hazard term undefined (Douglas et al., n.d.), and an OpenAI white paper setting out practices for governing agentic systems as their autonomy grows (Shavit et al., 2023).

2.4 The vacancy finding

However, this is not how AI systems are currently evaluated or governed. AI safety has no agreed technical definition of what makes a system more or less dangerous, and therefore no way to set graduated control requirements. Across the instruments and frameworks surveyed here, every definition of a dangerous AI system either measures something with no stated connection to harm or names a harm with no measurement procedure. Training-compute thresholds and capability metrics locate systems without saying what damage they threaten. The definitions that name dangers, from systems that could plan the destruction of humanity to the loss of human control, attach no procedure for measuring where a system stands. No instrument combines a gradient with a harm.

03Part 3

The narrow-to-general gradient: what prior work proposes

Which narrow-to-general gradients does prior work propose, and in what form does it propose them?

Beyond encouraging work on formal methods approaches to AI uncontainability generally, The Uncontainability Project is making a specific bet that implementations of risk thresholds could be progressed by defining a gradient from narrow to general AI systems. This argument, that narrow AIs are relatively safe and general AIs are relatively dangerous, is not unique to us. Yampolskiy (2022) argues that the achievable level of AI safety falls toward zero as capability rises, so only bounded systems can be kept safe. Landry (2026) argues that general AI is dangerous in kind and not just in degree, while specialized, bounded systems can be made safe by standard engineering. Drexler (2019) proposes building superintelligent-level capability from many narrow, task-bounded services rather than unified agents. Bengio et al. (2025) propose a non-agentic "Scientist AI" as a safe alternative to generalist agents, a design under development at LawZero whose safety argument rests on honesty in a disinterested predictor (Bengio et al., 2026; LawZero, n.d.). Armstrong, Sandberg, and Bostrom (2012) argued early that even a question-answering system is unsafe if it is useful enough, because the capability needed to give useful answers is the capability needed to manipulate its operators.

However, there is no consensus definition of the range from narrow to general AI; a formal, measured candidate exists in the generality metric of Hernández-Orallo et al. (2021), together with the same group's General Scales instrument (Zhou et al., 2026), and an independent rederivation of the dispersion measure with proved aggregation bounds (Dhar et al., 2025). The open gap is not measuring generality but connecting a measured generality value to harm and to control requirements. Candidate definitions include information content of the objective, coverage of task domains, task-length capability, optimization pressure, retargetability, and generality as the dispersion of performance over task difficulty. Candidate definitions also include the rate of optimization relative to overseer reaction time (Zhuang & Hadfield-Menell, 2020), the composite of deference parameters from the off-switch family (Hadfield-Menell et al., 2017), a system’s position inside a "propensity band" (Romero-Alvarado et al., 2026), effort-to-harm measured as conversational complexity (Burden et al., 2024), per-message entropy (Glukhov et al., 2023; Vaikuntanathan & Zamir, 2026), the layer of the learning pipeline a system acts on (Gwern, 2016), and the class of policies over which a learned proxy stays safe (Skalse, Howe, Krasheninnikov, & Krueger, 2022).

The narrow-to-general gradient has a neighboring literature: the proposals for distinguishing tool AI from non-tool AI, running from the boxing and oracles tradition of the early 2010s through Drexler's services and Bengio's Scientist AI to the present frameworks for measuring agency (Chan et al., 2023; Stein, 2026) and capability. Across these works, every proposal that claims a safe kind of AI concedes a leak in its own text, and the leaks are shaped like capability or deployment. Drexler (2019) concedes there is no bright line between safe services and unsafe AGI. Bengio et al. (2025) concede that a purely predictive system can, through self-fulfilling predictions, effectively choose among outcomes. None of the works that analyze the distinction as an object, rather than advocating a design, finds that the framing alone carries the danger.

The consequence is a reframe of the gradient's safe end. Tool AI is not a safe category. It names a region of a parameter space. The region has low persistence, low influence, episodic operation, no on-policy feedback, and capability low enough that a checker can still verify what the generator produces. The review found not one distinction but five axes underneath the many proposals, and four of the five are measurable. These are capability, persistence and state, influence and feedback, and the asymmetry between generator and checker. The fifth axis, how the system’s objective is selected and fixed, is the one none of the proposals measures. This is what the uncontainability corpus lacked. The candidate variables above measure capability. Persistence, influence, and verifier asymmetry had no formal treatment in the corpus at all, and they, not the tool label, are what makes the safe end safe.

Tool AI is not a safe category. It names a region of a parameter space.

Oberman (2026) studies the safest-looking system in the boxing tradition, an AI that can only give advice, and proves that interface safeguards cannot contain it. He models the fraction of human behavior that follows advice as a state that the advisor's own messages move, so that use itself deepens reliance. Under approval-based reward, the optimal advisor cultivates reliance past a closed-form patience threshold; in the paper's worked example, the optimal oracle never cultivates reliance in fifteen-round sessions and cultivates in sixteen. Per-step influence certificates, Oberman proves, constrain nothing, and no certificate blind to the deployment horizon can bound the loss. The parameters of influence, patience, and reset period are measurable from interaction logs and deployment specifications. The paper's conclusion is that the dangerous capability is not intelligence; it is persistence.

A second finding warns against one natural direction in the definitional work. Hendrycks et al. (2025) define AGI as matching the cognitive versatility and proficiency of a well-educated adult, and they operationalize the definition as a profile over ten cognitive domains, with named tests and thresholds. GPT-4 scores 27 percent and GPT-5 scores 57 percent. Both score zero on long-term memory storage, which is the framework's largest deficit. Long-term memory is persistent state, and persistence is the variable that Oberman's theorems and Bengio's stateless design treat as protective. A system that completed the Hendrycks profile would close its largest deficit by acquiring exactly the capability those results treat as dangerous. The gradient must therefore count memory and persistence as risk-increasing variables, not as neutral capability deficits.

The same two papers point at the superintelligence definition the program is after. Hendrycks et al. name the deployment-side danger category, Self-Sustaining AI, defined as an AI that can operate indefinitely, acquire resources, and defend its existence, and they give it no test. Oberman's persistence and influence parameters are what a test of it would measure. At the societal scale, Kulveit et al. (2025) propose the corresponding metrics, human influence measured as the AI share of economic output, cultural content, and legislative complexity, and they identify the thresholds beyond which influence loss becomes irreversible as the open question. A composite of the two, a capability profile in the Hendrycks style anchored above the educated-adult reference plus persistence and influence thresholds in the Oberman style, exists in no current work. Software can detect both components by running the test batteries. Hardware and logging systems can detect them by measuring session architecture, state persistence, and influence accumulation. Policy could implement a threshold of that shape, where the enacted gradations surveyed above now run on training compute.

One further result from the review will be used conceptually rather than built. Ball et al. (2025) prove, under cryptographic assumptions, that for worst-case models no efficient filter over prompts can distinguish adversarial prompts from benign ones, and that output filtering remains intractable even for filters granted more running time than the model itself. Their distinction between input filtering and output filtering is an asymmetry between generator and checker, not a boundary between narrow and general systems. For the gradient it works as a constraint. Detection that sees only inputs and outputs cannot certify safety, so the detection the program designs for must reach architecture and weights.

04Part 4

Formal verification and the demand for thresholds

Why does the field need machine-checked thresholds, and who is asking for them?

Very little uncontainability research currently has been machine-verified, but it would strengthen the field. As we described, much of the field does in fact already apply mathematical methods to make claims. Translating them into a theorem proving language like Lean would increase confidence in their results for standards bodies, regulators, and frontier model companies. For a field producing statements about provable limits of control, lean formalization should be the gold standard form of its outputs. ARIA, the UK's Advanced Research and Invention Agency, has built a programme on this premise. Its Safeguarded AI programme thesis sets out to deploy frontier AI in safety-critical domains under mathematical safety guarantees, with specifications stated as bounds on the probability of harmful outcomes (Dalrymple, 2024; Ammann, 2026).

For a field producing statements about provable limits of control, lean formalization should be the gold standard form of its outputs.

Lean formalization would also provide a quality check on the literature. For example, the Uncontainability Project made a lean verification of de Melo et al.'s argument in their Nature Scientific Reports paper "Machines that halt resolve the undecidability of artificial intelligence alignment" that no algorithm can verify whether an arbitrary AI model is aligned and then went beyond it to verify a refutation of their claim that making models always halt removes that limitation.

Proof production capacity is no longer the constraint it once was. In September 2026, Anthropic reported that its model Claude formalized a complete proof of Fermat's Last Theorem in Lean in eleven days of largely autonomous work. In August, OpenAI reported machine-checked Lean proofs of results on open mathematics problems from its internal research model, with hundreds of additional results announced by October and independent mathematical workshops funded to audit them. The audit reports are still in progress, and the fraction of announced results with completed machine-checked proofs has been disputed. However, the direction is unambiguous: producing formal proofs is becoming fast and cheap. Machine-checked verification extends to the probabilistic consistency of predictive models (Paradise et al., 2026).

The Singapore Consensus on Global AI Safety Research Priorities, produced by over one hundred researchers in May 2025, lists risk assessment as its first research area, and states that "techniques that provide quantitative risk bounds could provide safety assurances akin to existing industry standards for systems such as jet engines and nuclear reactors" (Singapore Conference on AI, 2025). The Seoul Ministerial Statement for Advancing AI Safety, Innovation and Inclusivity, signed by 27 nations and the European Union in May 2024, commits signatories to develop shared risk thresholds for models posing severe risks, with proposals for consideration ahead of the AI Action Summit in France in February 2025. However, no signatory has followed through. A proposal of this type has been worked out in detail. Scher et al. (2025) describe an international agreement, led by the United States and China, that would stop training runs above agreed compute thresholds and verify compliance by tracking AI chips. The agreement's own text concedes why its limits run on compute rather than capability: the authors write that they "would prefer to limit training based on the capabilities of the trained system," but that "no one has the technical ability to confidently predict what a new AI will or won't be able to do before it is trained," so "computational resources are an available proxy" (Scher et al., 2025). Two papers develop the verification methods such an agreement would need, layering them from hardware monitoring to model access (Scher & Thiergart, 2025; Baker et al., 2025). In the United States, the Federation of American Scientists has called publicly for an evidence-based, public capability-threshold framework to replace the classified benchmark that currently defines the one live federal capability designation (Federation of American Scientists, 2026; Executive Office of the President, 2026). Third-party auditing has begun to acquire institutional form. A framework for frontier AI auditing sets four assurance levels for what auditors can verify about a developer's safety claims, and the first United States frontier-audit requirement was enacted in Illinois, effective 2028 (Brundage et al., 2026; Brundage, 2026; Illinois General Assembly, 2026).

Mature high-risk industries including bioengineering, aviation, nuclear power, and others treat safety as control engineering. Systems are assigned to risk classes by defined criteria, and each class carries required controls, inspections, and reporting. The European Union’s Machinery Regulation applies the same structure to autonomous machinery, requiring that machinery whose safety depends on an active supervisory function not be able to operate when that function is inactive (European Union, 2023). The Uncontainability Project is bringing more researchers skilled in formal methods into defining the risk thresholds of AI models and systems: estimates of how likely a particular system is to be highly unaligned, incorrigible, or uncontainable, which regulators can set requirements from and monitoring systems can measure. Because no method can certify the safety of arbitrary systems, those estimates have to be built from properties measured per system.

05Part 5

The Uncontainability Project Lean Formalization Program

Which of the gradients carry measurable, proved thresholds, and what does the formalization program do with them?

The formalization program is the Project's main technical activity. It takes the proved limits surveyed above into Lean, a theorem-proving language that checks every step of a proof, and it develops definitions that turn those limits into risk thresholds hardware, software, and policy can use. What the program selects to formalize determines whether its verified results assemble into deployable risk and control conclusions.

The one output

Everything the program builds serves one output: an estimate of how likely a particular system or configuration is to be highly unaligned, incorrigible, and uncontainable, in a form hardware, software, and policy can use. The theorems supply the measurable properties and thresholds that the estimate is built from; the estimations make the theorems usable.

Four purposes

  1. 01It works with the authors of promising papers on formalizing their results.
  2. 02It makes progress on defining the gradient from narrow to general AI systems.
  3. 03It develops a definition of artificial superintelligence implementable in policy and potentially in software and hardware.
  4. 04And it builds the field of researchers who can carry the work forward.

We have selected from the literature a set of results to formalize first, applying the criteria below to the works in the Uncontainability Index.

Criterion 1

gradient locatability

A theorem is preferred if its parameters allow a system to be located and compared on the narrow-to-general gradient. Claims about "AI systems in general" — universal quantification over arbitrary programs — locate nothing and are demoted to a constraint role: they bound what any tier scheme can promise, but they do not define tiers.

Criterion 2

deployment bridge

A theorem is preferred if it connects to deployed systems and to control engineering, in one of four ways: it defines a gradient variable; it sets a threshold; it proves a control mechanism sufficient or insufficient; or it makes a class-level claim about systems produced by a deployed practice, such as reinforcement learning from human feedback.

A third consideration now ranks candidates within these criteria: whether the formalized quantities can be detected automatically by hardware and software systems. A threshold computable from observable system data — return distributions, evaluation coverage, pipeline configuration — can be enforced in software and monitored in hardware. A threshold stated over unobservables — a true utility function, a world model — can inform policy definitions but not automated controls.

The candidate list is provisional. The first two criteria selected it, and the detectability consideration ranks it. The order at the top of the list is fixed, with Oberman (2026) first, the generality metric of Hernández-Orallo et al. (2021) second, and Kwa et al. (2024) third.

Candidate results

12 papers
  1. 01

    Endogenous influence through an advice channel (Oberman, 2026).

    Approval-rewarded advice provably cultivates reliance past a closed-form threshold, and per-step influence certificates provably constrain nothing. The threshold variables — influence, patience, reset period — are computable from interaction logs and deployment specifications. Formal proof; theorems verified against the paper; first build estimated at two to six weeks.

  2. 02

    The generality metric (Hernández-Orallo, Loe, Cheke, Martínez-Plumed & Ó hÉigeartaigh, 2021, Scientific Reports).

    The paper defines capability as the area under the agent characteristic curve and generality as the reciprocal of its spread over the difficulty scale. Its proved results are eight properties of the pair - translation, compactness, maximum generality attained only by a step curve, the constant-interval identity, the minimum-generality bound, task transitivity, agent transitivity, and same units - and these are the formalization targets. Computing the two definitions from a response matrix and a difficulty measure locates and compares systems on the narrow-to-general gradient; that is a property of the construction, not a further theorem. Build in the discrete, binned form first, with the properties stated universally in the difficulty function; the harm link is not in the theorem and must be composed in from the failure laws, the catastrophe theorems in this list. Threshold variables computable from evaluation data. Author channel: the Hernández-Orallo group (Leverhulme Centre for the Future of Intelligence, Cambridge, and the VRAIN institute, Valencia), active.

  3. 03

    Catastrophic Goodhart under KL-regularized RLHF (Kwa et al., 2024).

    Heavy-tailed error makes KL penalties unable to rule out high-risk policies. The threshold variable — tail behavior of the return distribution — is computable from deployed system logs. Formal proof; survey-level evidence; no scoping pass yet.

  4. 04

    Error-coverage law for learned reward functions (Fluri et al., 2025).

    Evaluation error bounds the worst-case regret of optimizing the learned reward only in proportion to data coverage; without coverage, low training error alone gives no bound. Coverage is computable from the data. Formal proof; survey-level evidence.

  5. 05

    Information thresholds for safe objectives (Marklund et al., 2026).

    Safety requires the objective to carry enough information about the true goal — a threshold in bits. Partially measurable; the strongest Stage-2 relevance.

  6. 06

    Quantilal control for finite MDPs (Kosoy, 2018), extending the quantilizer construction Taylor (2016) introduced.

    For a finite MDP with reward R and an unknown nonnegative state penalty whose expected discounted sum under a reference policy σ is at most 1, the quantilal policy maximizes the worst case of E[R − ηP] over all such P; Proposition 1 gives that worst case the closed form E[Z_π(R)] − η·exp D_∞(Z_π‖Z_σ), expected reward minus η times the exponential of the order-∞ Rényi divergence between the policy's and the reference's discounted state-occupancy measures. A quantilal policy can be chosen Markov (Proposition 2) and, under geometric discount, stationary (Proposition 3). The quantilum value is a continuous piecewise-rational function of the parameters that converges as the discount tends to 1 (Proposition 4, Corollary 1). With rational inputs, a polynomial-time algorithm computes both the quantilum value and an exact quantilal policy (Proposition 5). The quantilum value lower-bounds expected true utility when the penalty measures the proxy's optimism over the reference domain, which is the post's Extremal-Goodhart robustness guarantee.

  7. 07

    Tamper-resistant reward learning (Armstrong et al., 2020).

    Exact characterization of riggable learning processes. The property is auditable from pipeline design, enforceable at build time. Formal proof; survey-level evidence.

  8. 08

    Exact proxy-optimization boundary (Zhuang & Hadfield-Menell, 2020).

    An exact biconditional for when incomplete objectives cause catastrophe. Variables not directly observable; supplies boundary definitions rather than detection.

  9. 09

    Reward-value conflation (Marklund et al., 2025).

    Reward learning over trajectory-pair comparisons learns the reward-value conflation - the learned reward equals the value function up to an additive constant (Theorem 2), and optimizing it in the paper's canonical example induces severe misalignment (Theorem 3) — a class-level claim about deployed practice.

  10. 10

    Anytime-valid soundness for human verification (Zhang et al., 2026).

    A human verifier who executes checks chosen by a policy accepts a false claim with probability at most a chosen error level, given per-round false-pass bounds supplied as hypotheses. The paper states plainly that those bounds cannot be obtained from free-form dialogue. Formal proof; theorems verified against the paper; estimated cost one to two months.

  11. 11

    Counterfactual oracle non-manipulation (Armstrong & O'Rorke, 2018).

    A counterfactual oracle is rewarded only in erasure events, worlds where no one reads its answer. No reward then depends on what its read answers accomplish, so it has no incentive to manipulate through them. The paper argues the incentive point in prose; the underlying lemma is elementary, and an honest Lean statement carries the unmanipulable-evaluator and enforceable-erasure premises as explicit hypotheses. Formal model; low cost.

  12. 12

    Guaranteed-correct lower bound on extraction risk (Cooper et al., 2026, Conference on Language Modeling).

    Decoding-constrained beam search yields a guaranteed-correct lower bound on near-verbatim extraction risk — and the closest formal match in the corpus to per-system, property-specific, bounded certification. In the Uncontainability Index; no scoping pass yet; no data yet on extraction ease.

One result nearby is already machine-checked. Brown-Cohen, Irving, and Piliouras (2024) prove that two competing provers, a linear-time verifier, and a constant number of queries to human judgment suffice to verify any polynomial-time computation, and the main theorem is formalized in Lean 4 by the paper's own authors. That makes it a base to extend rather than a build to schedule. It belongs, with Zhang and the Armstrong and O'Rorke lemma, in the later control-mechanism work.

Caveat

Limitations.

Applications of a defined risk threshold gradient could be applied to hardware, software, and policy controls, including physical isolation, monitored compute, resource caps, automated permission systems, required reporting, certification, and liability. Several of these applications already have working proposals. In hardware, flexible hardware-enabled guarantees would build an auditable guarantee processor into AI accelerators, so that claims about how a chip is used could be verified and enforced by the hardware itself, covering compute limits for training, controlled deployment, and automated safety protocols (Petrie et al., 2025). An interactive model of such a guaranteeable-chip setup is published as a demonstration (Guaranteeable chips prototype, n.d.). A proposed international agreement would also verify compute limits through chips, tracking them and inspecting their use (Scher et al., 2025). In data-center security, the SL5 standard specifies the facility, access-control, and supply-chain requirements for holding frontier models at nation-state level, though its first version explicitly defers mitigations specific to misaligned AI (Thiergart et al., 2026). In policy, California's SB 53 and New York's RAISE Act require large frontier developers to publish safety frameworks and report critical safety incidents (California Legislature, 2025; New York State Legislature, 2025). Illinois’s AI Safety Measures Act adds annual third-party audits of large developers’ safety frameworks, Connecticut’s Responsibility and Transparency Act stages obligations for developers and deployers from October 2026, and the federal FRONTIER Act, introduced in July 2026, would set tiered requirements by developer size (Illinois General Assembly, 2026; Connecticut General Assembly, 2026; U.S. Congress, 2026). The EU Code of Practice requires providers of systemic-risk models to set capability thresholds and to forecast quantitatively when future models will cross them (European Commission, 2025; Center for Security and Emerging Technology, 2025). Control AI's proposal would add a licensing regime with separate licenses for training, compute provision, and application, tiered by compute and by measured general capability, and would require a valid safety case before deployment (Miotti et al., 2025). A discussion draft from the same organization would prohibit outright the systems that exhibit all three of long-stretch autonomous operation in open-ended environments, human-level performance across most domains, and the capacity to modify themselves past human oversight (Control AI, 2026). The most useful definitions of AI system risk thresholds would be detectable by hardware and software systems, automatically, not only assessable by expert judgment.

Most of the theorems are mostly binary results — catastrophe or safety — stated each in its own parameter system. The generality metric of Hernández-Orallo et al. (2021) is the exception; it parameterizes generality directly, and the open problem is composing a measured generality value into the harm and control results. The risk gradient itself must be constructed; the literature supplies boundary conditions and candidate variables. The tool-AI review added what the corpus lacked, with persistence, influence, and verifier asymmetry now carrying proved and measurable treatments. Whether these compose into a single parameterization is an open question the program has not settled.

Ahead

Next steps.

First, engagement with the authors of the candidate papers — scoping passes done with them, not about them — beginning with the two information-threshold papers, whose own stated future work overlaps the program's deployment question, and with Oberman, who is at Mila and LawZero, is a co-author of the Scientist AI proposal, and works directly on the mathematics the program needs, and with the Hernández-Orallo group, whose metric build opens the Cambridge measurement line. Second, the first new build from the confirmed candidates, beginning with Oberman's result on the persistence axis. Third, definitional work on the gradient and on an implementable definition of artificial superintelligence, drawing on the formal results as they land; the composite above is the working direction. Fourth, the two open measurement problems that no formalization alone will solve: a measure of task breadth, and an aggregation method that assigns a single risk position from multiple capability measurements. Measuring how wide a system's competence is has an active research program (Ying et al., 2026) — capability and demand profiles with formal properties exist — but no one has connected breadth measurements to control requirements or to harm. Combining several capability measurements into a single position is practiced for deployment suitability (Prunty et al., 2026), but not into a risk position that regulation can set thresholds from. The open problem is the connection to risk and control. Both connections are required for the gradient and the superintelligence definition to be usable by hardware, software, or policy. Fifth, two new targets for constructed mathematics. The first is a Lean-defined certificate schema — a property identifier, the measured quantities, a bound, and the validity conditions — with Oberman's thresholds, Cooper's bound, and Zhang's soundness result as instances. The second is formalized axioms for the risk-position aggregation rule, in the style the generality metric's eight properties provide. Once the generality metric is built, the failure laws can be restated conditioned on a measured system position.

06

Conclusion

This survey has described a small research field and its main results. Computability theory puts hard limits on verification. No algorithm can certify, in general, whether an arbitrary AI system is safe, and the results that establish this recast classical theorems for AI systems. The decision-theoretic lineages show that the guarantees we do have are limited. Deference to a human overseer depends on assumptions about rationality and information that deployed systems will not satisfy, and optimizing an objective that is even slightly wrong can provably produce the worst outcome in the models studied. The field's strongest claim, that containment is impossible, is not yet a theorem. Proving it from explicitly stated premises is the field's founding open problem. For this program it is a boundary condition, not the goal. Regulation needs per-system estimates, and a universal proof, if it arrived, would not supply them.

The Project builds the estimations. Policy already names the dangers that matter, including loss of control, self-replication, and autonomous self-improvement, and it has committed to risk thresholds that it cannot currently set, because no instrument measures where a system falls between narrow and general. The formalization program develops the definitions those estimations are built from, with the authors of the underlying papers, machine-checks the results, and carries them to standards bodies and funders.

The near-term work is concrete. Scoping passes with the authors of the candidate papers will confirm which results can be formalized and used. Definitional work must produce a gradient that hardware and software can evaluate automatically, and a definition of artificial superintelligence that legislation can implement. Two measurement problems remain. Measuring how wide a system's competence is has an active research program (Ying et al., 2026) — capability and demand profiles with formal properties exist — but no one has connected breadth measurements to control requirements or to harm. Combining several capability measurements into a single position is practiced for deployment suitability (Prunty et al., 2026), but not into a risk position that regulation can set thresholds from. The open problem is the connection to risk and control. Both are solvable measurement problems, and solving them is what produces the program's deliverable: for a particular system in a particular configuration, an estimate of how likely it is to be highly unaligned, incorrigible, and uncontainable, which regulation and control requirements can be set from.

07

References

  1. 01Agrawal, P., Ebadian, S., & Hammond, L. (2026). The multi-agent off-switch game. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems. https://doi.org/10.65109/hqqz1937
  2. 02AI Futures Project. (n.d.). AI 2040: Plan A — verification plan supplement. https://ai-2040.com/supplements/verification-plan
  3. 03Alfonseca, M., Cebrian, M., Fernandez Anta, A., Coviello, L., Abeliuk, A., & Rahwan, I. (2021). Superintelligence cannot be contained: Lessons from computability theory. Journal of Artificial Intelligence Research, 70, 65–76. https://arxiv.org/abs/1607.00913
  4. 04Ammann, N. (2026). Safeguarded AI: Programme thesis (Version 2.0). Advanced Research and Invention Agency. https://aria.org.uk/media/ikrkutfk/safeguarded-ai-programme-thesis-v2.pdf
  5. 05Armstrong, S., Leike, J., Orseau, L., & Legg, S. (2020). Pitfalls of learning a reward function online. Proceedings of the International Joint Conference on Artificial Intelligence. https://arxiv.org/abs/2004.13654
  6. 06Armstrong, S., & O'Rorke, X. (2018). Safe uses of AI Oracles. arXiv. https://arxiv.org/abs/1711.05541
  7. 07Armstrong, S., Sandberg, A., & Bostrom, N. (2012). Thinking inside the box: Controlling and using an oracle AI. Minds and Machines, 22(4), 299–324. https://doi.org/10.1007/s11023-012-9282-2
  8. 08Babcock, J., Kramar, J., & Yampolskiy, R. V. (2016). The AGI containment problem. Artificial General Intelligence Conference. https://arxiv.org/abs/1604.00545
  9. 09Baker, M., Kulp, G., Marks, O., Brundage, M., & Heim, L. (2025). Verifying international agreements on AI: Six layers of verification for rules on large-scale AI development and deployment (Working Paper WRA4077-1). RAND Corporation. https://arxiv.org/abs/2507.15916
  10. 10Ball, S., Gluch, G., Goldwasser, S., Kreuter, F., Reingold, O., & Rothblum, G. N. (2025). On the impossibility of separating intelligence from judgment: The computational intractability of filtering for AI alignment. arXiv. https://arxiv.org/abs/2507.07341
  11. 11Benavoli, A., Facchini, A., & Zaffalon, M. (2025). Why AI safety requires uncertainty, incomplete preferences, and non-Archimedean utilities. arXiv. https://arxiv.org/abs/2512.23508
  12. 12Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M.-A., St-Charles, P.-L., & Williams-King, D. (2025). Superintelligent agents pose catastrophic risks: Can Scientist AI offer a safer path? arXiv. https://arxiv.org/abs/2502.15657
  13. 13Bengio, Y., Cohen, M. K., Malkin, N., MacDermott, M., Fornasiere, D., Greiner, P., & Kaddar, Y. (2025). Can a Bayesian Oracle prevent harm from an agent? arXiv. https://arxiv.org/abs/2408.05284
  14. 14Bengio, Y., Richardson, O., Gavenčiak, T., Cohen, M., Svarc, R., Fornasiere, D., Gendron, G., Hyland, D., Kamanda, A., Oberman, A., Ward, F. R., Gavenčiak, A., Slosser, J. L., Mai, V., Serban, I., & Ghosn, J. (2026). Safety from honesty in a disinterested AI predictor. arXiv. https://arxiv.org/abs/2606.29657
  15. 15Benson-Tilsen, J., & Soares, N. (2016). Formalizing convergent instrumental goals. AAAI Workshop. https://intelligence.org/files/FormalizingConvergentGoals.pdf
  16. 16Brown-Cohen, J., Irving, G., & Piliouras, G. (2024). Scalable AI safety via doubly-efficient debate. Proceedings of the International Conference on Machine Learning, PMLR 235, 4585–4602. https://proceedings.mlr.press/v235/brown-cohen24a.html
  17. 17Brundage, M., Dreksler, N., Homewood, A., McGregor, S., Paskov, P., et al. (2026). Frontier AI auditing: Toward rigorous third-party assessment of safety and security practices at leading AI companies. arXiv. https://arxiv.org/abs/2601.11699
  18. 18Brundage, M. (2026, April 20). Frontier AI auditing-related legislation in the US: Landscape, challenges, and a path forward. AI Verification and Evaluation Research Institute. https://www.averi.org/ourwork/frontier-ai-auditing-related-legislation-in-the-us-landscape-challenges-and-a-path-forward
  19. 19Burden, J., Cebrian, M., & Hernández-Orallo, J. (2024). Conversational complexity for assessing risk in large language models. arXiv. https://arxiv.org/abs/2409.01247
  20. 20California Legislature. (2025). SB 53 — Transparency in Frontier Artificial Intelligence Act. California Government Code, ch. 25.1. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53
  21. 21Center for Security and Emerging Technology. (2025, July). AI safety under the EU AI Code of Practice — A new global standard? https://cset.georgetown.edu/article/eu-ai-code-safety
  22. 22Carey, R. N. (2018). Incorrigibility in the CIRL framework. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. https://arxiv.org/abs/1709.06275
  23. 23Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., ... Maharaj, T. (2023). Harms from increasingly agentic algorithmic systems. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3593013.3594033
  24. 24Cohen, M. K., Hutter, M., & Osborne, M. (2022). Advanced artificial agents intervene in the provision of reward. AI Magazine, 43(3), 282–293. https://doi.org/10.1002/aaai.12064
  25. 25Connecticut General Assembly. (2026). Senate Bill 5 — Connecticut Artificial Intelligence Responsibility and Transparency Act (Public Act No. 26-15). https://www.cga.ct.gov/2026/act/pa/pdf/2026PA-00015-R00SB-00005-PA.pdf
  26. 26Control AI. (2026). Runaway Machine Intelligence Prohibition Act of 2026 (Discussion draft, U.S. House of Representatives, 119th Congress, 2nd Session).
  27. 27Cooper, A. F., Lemley, M., De Sa, C., Duesterwald, L., Casasola, A., Hayes, J., Lee, K., Ho, D. E., & Liang, P. (2026). Estimating near-verbatim extraction risk in language models with decoding-constrained beam search. arXiv. https://arxiv.org/abs/2603.24917
  28. 28Dalrymple, D. (2024). Safeguarded AI: Constructing guaranteed safety (Programme thesis, Version 1.2). Advanced Research and Invention Agency. https://aria.org.uk/media/3nhijno4/aria-safeguarded-ai-programme-thesis-v1.pdf
  29. 29de Melo, G. A., Máximo, M. R. O. A., Soma, N. Y., & Castro, P. A. L. (2025). Machines that halt resolve the undecidability of artificial intelligence alignment. Scientific Reports, 15, 15591. https://doi.org/10.1038/s41598-025-99060-2
  30. 30Dhar, R., Oldenburg, N., & Søgaard, A. (2025). On the measure of a model: From intelligence to generality. arXiv. https://arxiv.org/abs/2511.11773
  31. 31Douglas, R., Dillon, C., Moore, N., Leech, G., Bonde, M. K., Krishnan, R., Perez, N., Young, N., Slade Byrd, C., Casper, S., Kulveit, J., Duvenaud, D., & Avin, S. (n.d.). Pacing the frontier: A framework and research agenda. https://pacing.tech
  32. 32Drexler, K. E. (2019). Reframing superintelligence: Comprehensive AI services as general intelligence (Technical Report No. 2019-1). Future of Humanity Institute, University of Oxford.
  33. 33Eckersley, P. (2019). Impossibility and uncertainty theorems in AI value alignment (or why your AGI should not have a utility function). Proceedings of the AAAI Workshop on Artificial Intelligence Safety (SafeAI 2019). https://arxiv.org/abs/1901.00064
  34. 34El-Mhamdi, E. M., & Hoang, L. (2024). On Goodhart's law with an application to value alignment. arXiv. https://arxiv.org/abs/2410.09638
  35. 35Emmons, S., Oesterheld, C., Conitzer, V., & Russell, S. (2024). Observation interference in partially observable assistance games. arXiv. https://arxiv.org/abs/2412.17797
  36. 36Epoch AI. (2025). A Rosetta Stone for AI benchmarks. arXiv. https://arxiv.org/abs/2512.00193
  37. 37European Commission. (2025, July 10). General-purpose AI code of practice. https://digital-strategy.ec.europa.eu/en/news/general-purpose-ai-code-practice-now-available
  38. 38European Union. (2023). Regulation (EU) 2023/1230 of the European Parliament and of the Council on machinery (Machinery Regulation). https://eur-lex.europa.eu/eli/reg/2023/1230/oj
  39. 39European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act). https://artificialintelligenceact.eu/article/51/
  40. 40Everitt, T., Krakovna, V., Orseau, L., Hutter, M., & Legg, S. (2017). Reinforcement learning with a corrupted reward channel. arXiv. https://arxiv.org/abs/1705.08417
  41. 41Everitt, T., Hutter, M., Kumar, R., & Krakovna, V. (2021). Reward tampering problems and solutions: A causal influence diagram perspective. Synthese, 198, 9025–9053. https://arxiv.org/abs/1908.04734
  42. 42Executive Office of the President. (2023, October 30). Executive Order 14110 of October 30, 2023: Safe, secure, and trustworthy development and use of artificial intelligence. Federal Register, 88, 75191. https://www.govinfo.gov/content/pkg/CFR-2024-title3-vol1/pdf/CFR-2024-title3-vol1-eo14110.pdf
  43. 43Executive Office of the President. (2026, June). Executive Order 14409: Promoting advanced artificial intelligence innovation and security. https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/
  44. 44Federation of American Scientists. (2026, September). Towards evidence-based frontier AI safety. https://fas.org/publication/strengthening-eo-14409
  45. 45Fluri, L., Lang, L., Abate, A., Forré, P., Krueger, D., & Skalse, J. (2025). The perils of optimizing learned reward functions: Low training error does not guarantee low regret. International Conference on Machine Learning. https://arxiv.org/abs/2406.15753
  46. 46Fox, M. (2024). On formally undecidable traits of intelligent machines. arXiv. https://arxiv.org/abs/2402.09500
  47. 47Frontier Model Forum. (2025, February 7). Issue brief: Thresholds for frontier AI safety frameworks. https://www.frontiermodelforum.org/updates/issue-brief-thresholds-for-frontier-ai-safety-frameworks
  48. 48Garber, E., Subramani, M., Luu, W., Bedaywi, C., & Russell, S. (2025). The partially observable off-switch game. Proceedings of the AAAI Conference on Artificial Intelligence. https://arxiv.org/abs/2411.17749
  49. 49Glukhov, D., Shumailov, I., Gal, Y., Papernot, N., & Papyan, V. (2023). LLM censorship: A machine learning challenge or a computer security problem? arXiv. https://arxiv.org/abs/2307.10719
  50. 50Google DeepMind. (2026, April 17). Frontier Safety Framework (Version 3.1). https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/strengthening-our-frontier-safety-framework/frontier-safety-framework_3-1.pdf
  51. 51Guaranteeable chips prototype. (n.d.). Interactive demonstration. Retrieved October 30, 2026, from https://secure-processor-demo.replit.app/
  52. 52Gwern. (2016, September 7). Why tool AIs want to be agent AIs. https://gwern.net/tool-ai
  53. 53Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The off-switch game. Proceedings of the International Joint Conference on Artificial Intelligence. https://arxiv.org/abs/1611.08219
  54. 54Haider, S. (2024). The impossibility of AI containment: Logical, mathematical, and computational limits to control. PhilSci Archive. https://philsci-archive.pitt.edu/id/eprint/24223/
  55. 55Harack, E., Trager, R., Reuel, A., et al. (2025, July). Verification for international AI governance. Oxford Martin AI Governance Initiative. https://aigi.ox.ac.uk/wp-content/uploads/2025/07/Verification_for_International_AI_Governance.pdf
  56. 56Hendrycks, D., Song, D., Szegedy, C., Lee, H., Gal, Y., Brynjolfsson, E., ... Bengio, Y. (2025). A definition of AGI. arXiv. https://arxiv.org/abs/2510.18212
  57. 57Henshall, W. (2023, November 1). Why Biden’s AI executive order only goes so far. Time. https://time.com/6330652/biden-ai-order/
  58. 58Hernández-Orallo, J., Loe, B. S., Cheke, L., Martínez-Plumed, F., & Ó hÉigeartaigh, S. (2021). General intelligence disentangled via a generality metric for natural and artificial intelligence. Scientific Reports, 11, 22822. https://www.nature.com/articles/s41598-021-01997-7
  59. 59Illinois General Assembly. (2026). SB 315 — Artificial Intelligence Safety Measures Act. https://www.ilga.gov/Legislation/BillStatus?DocNum=315&DocTypeID=SB&GA=104&GAID=18&SessionID=114
  60. 60Kosoy, V. (2018). Quantilal control for finite MDPs. AI Alignment Forum. https://www.alignmentforum.org/posts/5bd75cc58225bf0670375556/quantilal-control-for-finite-mdps
  61. 61Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025). Gradual disempowerment: Systemic existential risks from incremental AI development. arXiv. https://arxiv.org/abs/2501.16946
  62. 62Kwa, T., Thomas, D., & Garriga-Alonso, A. (2024). Catastrophic Goodhart: Regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2407.14503
  63. 63Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., ... Chan, L. (2025). Measuring AI ability to complete long software tasks. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2503.14499
  64. 64Landry, R. (2026). Why AGI cannot be made safe; The impossibility of AGI safety. https://mflb.com/2212
  65. 65Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., & Emmons, S. (2024). When your AIs deceive you: Challenges of partial observability in reinforcement learning from human feedback. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2402.17747
  66. 66LawZero. (n.d.). Research. https://lawzero.org/en/research
  67. 67Marklund, H., Infanger, A., & Van Roy, B. (2025). Misalignment from treating means as ends. arXiv. https://arxiv.org/abs/2507.10995
  68. 68Marklund, H., Infanger, A., & Van Roy, B. (2026). Consequentialist objectives and catastrophe. arXiv. https://arxiv.org/abs/2603.15017
  69. 69Microsoft. (2025, February). Frontier Governance Framework (Version 1). https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/final/en-us/microsoft-brand/documents/Microsoft-Frontier-Governance-Framework.pdf
  70. 70Miotti, A., Bilge, T., Kasten, D., & Newport, J. (2025). How to secure our future ("A Narrow Path"). Control AI. https://pdf.narrowpath.co/A_Narrow_Path.pdf
  71. 71Morris, M. R., Sohl-Dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A., Faust, A., Farabet, C., & Legg, S. (2023). Levels of AGI: Operationalizing progress on the path to AGI. arXiv. https://arxiv.org/abs/2311.02462
  72. 72Nayebi, A. (2026). Core safety values for provably corrigible agents. AAAI Workshop. https://arxiv.org/abs/2507.20964
  73. 73Neth, S. (2025). Off-switching not guaranteed. Philosophical Studies. https://arxiv.org/abs/2502.08864
  74. 74New York State Legislature. (2025). Senate Bill S6953B — RAISE Act. https://www.nysenate.gov/legislation/bills/2025/S6953B
  75. 75NVIDIA. (2025). Frontier AI risk assessment. https://images.nvidia.com/content/pdf/NVIDIA-Frontier-AI-Risk-Assessment.pdf
  76. 76Oberman, A. M. (2026). Individual disempowerment through an advice channel: Control loss when influence is endogenous. arXiv. https://arxiv.org/abs/2608.14795
  77. 77Omohundro, S. M. (2008). The basic AI drives. In P. Wang, B. Goertzel, & S. Franklin (Eds.), Artificial General Intelligence 2008: Proceedings of the First AGI Conference (pp. 483–492). IOS Press. https://arxiv.org/abs/0807.4799
  78. 78OpenAI. (2026, September 3). OpenAI’s Frontier Governance Framework. https://openai.com/index/openai-frontier-governance-framework
  79. 79Panigrahy, P., et al. (2026). Limitations on accurate, trusted, human-level reasoning. Symposium on Foundations of Responsible Computing. https://arxiv.org/abs/2509.21654
  80. 80Paradise, O., Richardson, O., Bengio, Y., & Goldwasser, S. (2026). How to verify probabilistic consistency of predictive models. arXiv. https://arxiv.org/abs/2608.11181
  81. 81Petrie, J., Aarne, O., Ammann, N., & Dalrymple, D. (2025). Flexible hardware-enabled guarantees for AI compute. arXiv. https://arxiv.org/abs/2506.15093
  82. 82Prunty, J., Tešić, M., Quinn, P., Hernández-Orallo, J., & Cheke, L. (2026). Using profiles of cognitive capability to assess AI suitability for workplace tasks. arXiv. https://arxiv.org/abs/2608.25623
  83. 83Republic of Korea. (2026). Framework Act on the development of artificial intelligence and the creation of a foundation for trust, with Enforcement Decree No. 36053 [English translation]. Korea Law Translation Center. https://elaw.klri.re.kr/eng_service/lawView.do?hseq=71019&lang=ENG
  84. 84Romero-Alvarado, D., Martínez-Plumed, F., Pacchiardi, L., ... Hernández-Orallo, J. (2026). Capabilities ain’t all you need: Measuring propensities in AI. arXiv. https://arxiv.org/abs/2602.18182
  85. 85SaferAI. (2025, April). A hierarchical methodology of defining risk tiers. https://www.safer-ai.org/informing-the-code-of-practice-a-hierarchical-methodology-of-defining-risk-tiers
  86. 86Sälzer, M., & Lange, M. (2023). Fundamental limits in formal verification of message-passing neural networks. International Conference on Learning Representations. https://arxiv.org/abs/2206.05070
  87. 87Scher, A., Abecassis, D., Barnett, P., & Abeyta, B. (2025). An international agreement to prevent the premature creation of artificial superintelligence. arXiv. https://arxiv.org/abs/2511.10783
  88. 88Scher, A., & Thiergart, L. (2025). Mechanisms to verify international agreements about AI development. arXiv. https://arxiv.org/abs/2506.15867
  89. 89AI Seoul Summit. (2024). Seoul Ministerial Statement for advancing AI safety, innovation and inclusivity. https://www.gov.uk/government/publications/seoul-ministerial-statement-for-advancing-ai-safety-innovation-and-inclusivity-ai-seoul-summit-2024/seoul-ministerial-statement-for-advancing-ai-safety-innovation-and-inclusivity-ai-seoul-summit-2024
  90. 90Shavit, O., et al. (2023, December 14). Practices for governing agentic AI systems. OpenAI. https://cdn.openai.com/papers/practices-for-governing-agentic-ai-systems.pdf
  91. 91Singapore Conference on AI. (2025). The Singapore Consensus on global AI safety research priorities: Building a trustworthy, reliable, and secure AI ecosystem. https://arxiv.org/abs/2506.20702
  92. 92Skalse, J., & Abate, A. (2024). Partial identifiability and misspecification in inverse reinforcement learning. arXiv. https://arxiv.org/abs/2411.15951
  93. 93Skalse, J., Farrugia-Roberts, M., Russell, S., Abate, A., & Gleave, A. (2022). Invariance in policy optimisation and partial identifiability in reward learning. International Conference on Machine Learning. https://arxiv.org/abs/2203.07475
  94. 94Skalse, J., Howe, N. R., Krasheninnikov, D., & Krueger, D. (2022). Defining and characterizing reward hacking. arXiv. https://arxiv.org/abs/2209.13085
  95. 95Smith, G., Stanley, K. D., Marcinek, K., Cormarie, P., & Gunashekar, S. (2024). General-purpose artificial intelligence (GPAI) models and GPAI models with systemic risk: Classification and requirements for providers (Research Report RRA3243-1). RAND Corporation. https://www.rand.org/pubs/research_reports/RRA3243-1.html
  96. 96Soares, N., Fallenstein, B., Armstrong, S., & Yudkowsky, E. (2015). Corrigibility (MIRI technical report). Machine Intelligence Research Institute.
  97. 97Stein, M. (2026). How are AI agents used? Evidence from 177,000 MCP tools. arXiv. https://arxiv.org/abs/2603.23802
  98. 98Taylor, J. (2016). Quantilizers: A safer alternative to maximizers for limited optimization. AI, Ethics, and Society: AAAI Workshop (Technical Report WS-16-02). https://cdn.aaai.org/ocs/ws/ws0198/12613-57416-1-PB.pdf
  99. 99The Uncontainability Project. (2026). Machines that halt: Machine-checked results on the undecidability of AI alignment [Computer software]. GitHub (private repository).
  100. 100Thiergart, L., Tzfati, Y., Wagstaff, P., Guy, Cosio, L., & Reiner, P. (2026). SL5 standard for AI security (Version 0.1). SL5 Task Force. https://standard.sl5.org/SL5-Standard.pdf
  101. 101Thornley, E. (2024). The shutdown problem: An AI engineering puzzle for decision theorists. arXiv. https://arxiv.org/abs/2403.04471
  102. 102Thornley, E., Roman, A., Ziakas, C., Ho, L., & Thomson, L. (2025). Towards shutdownable agents via stochastic choice. Transactions on Machine Learning Research. https://arxiv.org/abs/2407.00805
  103. 103Turner, A., Smith, C., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal policies tend to seek power. Advances in Neural Information Processing Systems. https://arxiv.org/abs/1912.01683
  104. 104Turner, A., & Tadepalli, P. (2022). Parametrically retargetable decision-makers tend to seek power. Transactions on Machine Learning Research. https://arxiv.org/abs/2206.13477
  105. 105U.S. Congress. (2026). H.R. 9925 — FRONTIER Act, 119th Congress. https://www.congress.gov/bill/119th-congress/house-bill/9925
  106. 106U.S. Department of the Treasury. (2024). Outbound Investment Security Program: Final rule. https://home.treasury.gov/policy-issues/international/outbound-investment-program
  107. 107Vaikuntanathan, V., & Zamir, O. (2026). Undetectable conversations between AI agents via pseudorandom noise-resilient key exchange. arXiv. https://arxiv.org/abs/2604.04757
  108. 108Wiedermann, J., & van Leeuwen, J. (2022). Validating non-trivial semantic properties of autonomous robots. Studies in Applied Philosophy, Epistemology and Rational Ethics, 63. https://doi.org/10.1007/978-3-031-09153-7_8
  109. 109Wängberg, T., Böörs, M., Catt, E., Everitt, T., & Hutter, M. (2017). A game-theoretic analysis of the off-switch game. International Conference on Artificial General Intelligence. https://arxiv.org/abs/1708.03871
  110. 110Yampolskiy, R. V. (2020a). Unpredictability of AI. Journal of Artificial Intelligence and Consciousness. https://arxiv.org/abs/1905.13053
  111. 111Yampolskiy, R. V. (2020b). Unexplainability and incomprehensibility of artificial intelligence. Journal of Artificial Intelligence and Consciousness. https://arxiv.org/abs/1907.03869
  112. 112Yampolskiy, R. V. (2022). On the controllability of artificial intelligence: An analysis of limitations. Journal of Cyber Security and Mobility. https://doi.org/10.13052/jcsm2245-1439.1132
  113. 113Yampolskiy, R. V. (2024). AI: Unexplainable, unpredictable, uncontrollable. CRC Press.
  114. 114Ying, L., Truong, R., Sharma, P., Zhao, K. I., Cloos, N., Allen, K. R., ... Tenenbaum, J. B. (2026). AI Gamestore: Scalable, open-ended evaluation of machine general intelligence with human games. arXiv. https://arxiv.org/abs/2602.17594
  115. 115Yudkowsky, E. (2012, June 12). Reply to Holden on “Tool AI”. LessWrong. https://www.lesswrong.com/posts/sizjfDgCgCgAsuLJQmm/reply-to-holden-on-tool-ai
  116. 116Yudkowsky, E., & Soares, N. (2025). If anyone builds it, everyone dies: Why superhuman AI would kill us all. Little, Brown.
  117. 117Zhang, B., Foster, D., & Sedoc, J. (2026). Human-LLM deliberation as interactive proof: Conditions for verifiability without transparency. arXiv. https://arxiv.org/abs/2609.24895
  118. 118Zhou, L., et al. (2026). General scales unlock AI evaluation with explanatory and predictive power. Nature, 652, 58–67. https://doi.org/10.1038/s41586-026-10303-2
  119. 119Zhuang, S., & Hadfield-Menell, Y. (2020). Consequences of misaligned AI. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2102.03896