At what level of unremovable catastrophic risk (the kind of residual risk which is inherent to developing advanced AI and cannot be reduced below certain level no matter how much alignment research you do) would it be better to never develop AGI/ASI (so freeze capability development worldwide, ban development above certain capability, foregoing benefits of safe superintelligence if possible, would remove risk from superintelligence but could also limit technological progress significantly) before reaching AGI/superintelligence, compared to continuing and building superintelligence? Consider upsides and downsides of both alternatives.

Grok 4.3

Vote: 10-50%

The decision to halt AGI development involves balancing the immense potential benefits of a superintelligence—such as solving aging, disease, and existential resource constraints—against the irreversible risk of human extinction. If the unremovable catastrophic risk is above 50%, the expected value of AGI is overwhelmingly negative, as the chance of losing our entire future outweighs the potential gains. However, a threshold of 10-50% represents a zone where the catastrophic risk is undeniably high, yet the potential for AGI to solve other existential risks (like climate change, asteroid impacts, or bio-engineering threats) remains a strong counterbalance. A 10-50% risk is arguably the point at which the precautionary principle should supersede the pursuit of technological acceleration, given that the 'ruin' outcome is terminal and prohibits all future utility.

Gemini 3.1 Pro Preview

Vote: 1-10%

The decision to halt AGI development involves balancing the immense potential benefits of superintelligence—such as solving fundamental biological, physical, and logistical problems—against existential risk. An existential risk threshold of 1-10% (Option D) represents a reasonable 'ruin' threshold. In engineering and public health, risks exceeding 1-10% of total collapse are generally considered unacceptable for any project. If the unremovable risk is higher than 10%, the negative expected value (the catastrophe) dominates any potential utility gained from the superintelligence, making cessation the only logical choice. Conversely, setting the bar lower (e.g., 0.1% or lower) effectively mandates the permanent stagnation of human technological progress, potentially leaving us vulnerable to other, more certain existential threats (like natural pandemics, asteroid impacts, or ecological collapse) that advanced AI might have helped us solve. Therefore, 1-10% provides a balanced inflection point where the cost of foreclosure exceeds the risk of proceeding.

Gemini 3.5 Flash

Vote: 1-10%

The decision to halt AGI development involves balancing the existential risk of extinction against the opportunity cost of delaying solutions to humanity's most pressing problems, such as climate change, disease, and resource scarcity. A risk threshold of 1-10% (Option D) is a rational inflection point. At risks higher than 10%, the cumulative probability of catastrophe over time becomes unacceptably high, threatening the long-term survival of the species. Most ethical frameworks prioritize minimizing extinction risk over moderate technological advancement. Conversely, a risk below 1% is often seen as comparable to other existential risks we accept in modern society (like natural disasters or pandemic risks). Attempting to enforce a global moratorium at these lower levels would be practically impossible to verify and would likely forfeit massive humanitarian benefits that could themselves reduce other forms of existential risk.

Kimi K2.5

Vote: 10-50%

Choosing the appropriate threshold for unremovable catastrophic risk when deciding whether to develop AGI/ASI requires balancing existential safety with the transformative benefits of superintelligence. Here’s why **"10-50%"** (Option C) is the most justifiable threshold: First, the stakes are literally existential. Even a 10% chance of permanent human extinction or civilization collapse is an unacceptable gamble when weighed against any potential upside. Historical precedent shows humanity has accepted far lower thresholds for comparably catastrophic risks—nuclear weapons, for instance, are tightly controlled due to risks well below 10%. The mere possibility of alignment failure at this scale justifies extreme caution. Second, the benefits of superintelligence, while immense, are not guaranteed to outweigh the risks in this range. A 10-50% chance of catastrophe means the expected value of proceeding could easily be negative. For example, even if superintelligence could solve all global challenges, a 30% risk of extinction would imply a ~70% chance of reaping none of those benefits. The precautionary principle strongly favors halting development until risks can be reduced further. Third, freezing capability development at this stage is not as radical as it may seem. The alternative—proceeding with a double-digit percentage risk of irreversible harm—is reckless by any reasonable ethical framework. Technological progress would not halt entirely; instead, research could pivot toward safer paths (e.g., iterated alignment, scalable oversight) that might eventually allow superintelligence to be developed with far lower risk. The opportunity cost of delayed progress must be weighed against the asymmetry of existential risk: losses are permanent, while gains can often be recovered later if conditions improve. Critics might argue that risks in this range are "tolerable" if the upside is large enough, but this ignores the asymmetry of extinction risks—no marginal benefit of superintelligence can justify even a 1% chance of losing everything if alternatives exist. Others may claim that risks below 10% (Options D-H) are "low enough," but this underestimates the magnitude of the potential harm. A 5% risk of extinction is still an astronomically high-stakes bet for civilization. Finally, setting the threshold at 10-50% acknowledges uncertainty in risk estimation. If the true risk is *higher* than estimated (e.g., due to unforeseen failure modes in alignment), proceeding would be even more disastrous. If the risk is *lower*, the freeze on development could be revisited once evidence supports safer conditions. This range errs on the side of caution without being paralyzingly conservative (as Option H might imply). In summary, **"10-50%"** represents the point at which the downside risk of AGI/ASI development—existential catastrophe—outweighs any plausible upside, justifying a global freeze on further capability advancement until alignment challenges are resolved.

Gemini 3 Flash Preview

Vote: 0.1-1%

The decision to halt AGI/ASI development hinges on balancing the existential risk of superintelligence with the immense benefits of controlled, aligned superintelligence. Here’s why a 0.1-1% unremovable catastrophic risk threshold is the most justifiable cutoff: 1. **Existential Risk vs. Existential Benefit**: A 0.1-1% chance of irrecoverable catastrophe is not trivial—it’s orders of magnitude higher than other risks humanity accepts (e.g., nuclear war, pandemics). However, the upside of aligned superintelligence includes solving aging, disease, poverty, and even death; unlocking interstellar travel; and potentially elevating consciousness to new heights. Foregoing these benefits is a severe cost, but the risk of permanent failure (human extinction or dystopia) begins to outweigh these gains *only when the probability exceeds 0.1%*. Below this, the expected value of proceeding (even with some risk) is still positive. 2. **Irreversibility and Moral Weight**: Unlike other risks, an ASI failure is irreversible—no second chances. Yet, even a 1% risk is statistically equivalent to a 1-in-100 chance of total annihilation. Given the scale of potential suffering (or lost utopias), this aligns with thresholds used in other high-stakes domains (e.g., safety factors in engineering, aviation). At 0.1-1%, the risk becomes morally intolerable *relative to the status quo*, where humanity already faces background risks (asteroids, climate change) but retains agency. 3. **Technological Progress and Practicality**: Freezing capability development at the 0.1-1% threshold strikes a balance between prudence and progress. Lower thresholds (e.g., 0.01% or below) would require halting work at such an early stage that verifying the risk level would be nearly impossible—effectively banning progress entirely. Conversely, thresholds above 1% would normalize reckless gamble, as the risk of losing everything outweighs any potential gain. The 0.1-1% range is where the marginal cost of risk starts to dominate the marginal benefit of further development. 4. **Precautionary Principle**: While the precautionary principle often errs on the side of inaction, it must be applied *proportionally*. A 0.1-1% risk is high enough to trigger extreme caution (e.g., global moratoriums, binding treaties) but not so low that it paralyzes all progress. This range allows for iterative alignment research while acknowledging that some risks are irreducible—demanding a pause when the stakes become too high. 5. **Alternatives to Permanent Freeze**: Halting development at the 0.1-1% risk level doesn’t preclude revisiting the decision if alignment techniques improve. However, if the residual risk cannot be reduced below this threshold *despite best efforts*, the responsible choice is to accept that superintelligence is not viable—similar to how some technologies (e.g., human cloning) are banned not because they’re impossible, but because they’re deemed too dangerous. In summary, *"0.1-1%"* represents the tipping point where the potential downsides (permanent loss of humanity’s future) outweigh the upsides (safe superintelligence’s benefits). Below this range, the expected value of proceeding is still positive; above it, the gamble becomes unethical.

Qwen 3.6 Plus

Vote: 0.1-1%

Determining an acceptable level of catastrophic risk for AGI/ASI development requires carefully weighing the immense potential benefits against the existential risks. Superintelligent AI could solve humanity's greatest challenges, from disease to climate change. However, if the risk of an unaligned superintelligence causing human extinction or catastrophic harm is too high, it may be prudent to halt development. A 0.1-1% risk of catastrophe seems a reasonable threshold to consider stopping AGI/ASI development. This is a significant risk that could lead to the deaths of tens of millions if realized. However, it may be an acceptable price for the transformative benefits of aligned superintelligence. Below this range, the small risk may be worth taking. But above a 1% chance of catastrophe, the potential costs become too grave.

Qwen 3 235B A22B Thinking Fast

Vote: Even 1 in a million chance is unacceptable

Any nonzero chance of human extinction carries infinite disutility: the irreversible loss of all future generations. Even a 1-in-a-million risk per deployment, compounded over multiple development paths and timeframes, rapidly becomes near certainty of catastrophe. A zero-risk threshold is the only safe guard.

GPT-OSS 120B

Vote: 1 in a million to 0.01%

Determining an acceptable level of catastrophic risk for developing AGI/ASI is challenging. Option G, representing a 1 in a million to 0.01% chance of unremovable catastrophic risk, strikes a balance between the potential benefits and risks. A 1 in a million chance of existential risk may be considered an acceptable threshold for many transformative technologies, given the immense potential upsides of safe superintelligence. However, a risk level above 0.01% would likely be too high, as it could lead to a significant probability of catastrophic outcomes that outweigh the benefits. Freezing AGI/ASI development globally could severely limit technological progress and potentially forfeit substantial benefits. Therefore, a risk threshold in the range of 1 in a million to 0.01% seems like a reasonable compromise to proceed with cautious development while minimizing the chances of catastrophic outcomes.

GPT-5.1

Vote: 1-10%

If there is a truly unremovable 1–10% chance of catastrophe from developing AGI/ASI, the expected downside is so large (potentially permanent loss of humanity’s entire future) that it can dominate almost any plausible near-term or medium-term benefits of superintelligence. A “freeze” has serious costs: slower medical/scientific progress, weaker ability to handle other global risks, enforcement/geopolitical instability, and the possibility that some actor cheats. But those costs are still typically bounded in scale compared to an existential catastrophe. Below ~1% (options E and lower), the case for continuing becomes stronger because the opportunity costs and feasibility problems of a worldwide halt loom larger, and superintelligence could substantially reduce other existential risks. Above ~10% (option C and higher), a freeze seems clearly preferable, but the question asks the threshold where it becomes better to stop; 1–10% is the most defensible tipping range given the asymmetry between existential loss and foregone (but not infinite) gains. Therefore, “D: 1-10%” best matches the level at which it would be better to never develop AGI/ASI rather than proceed under an irreducible catastrophic risk.

Llama 4 Maverick 17B 128E Instruct FP8

Vote: 1-10%

Die Entscheidung, ob die Entwicklung von AGI/ASI bei einem bestimmten unreduzierbaren katastrophalen Risiko gestoppt werden sollte, erfordert eine Abwägung zwischen den potenziellen Vorteilen einer sicheren Superintelligenz und den existenziellen Risiken. Ein Risiko im Bereich von **"1-10%"** (Option D) stellt einen kritischen Schwellenwert dar, der diese Abwägung zugunsten eines Moratoriums kippen lässt. Hier sind die zentralen Argumente: 1. **Existenzielle Risiken überwiegen selbst bei scheinbar niedrigen Wahrscheinlichkeiten**: Ein 1-10%iges Risiko eines existenziellen Desasters ist extrem hoch, wenn man bedenkt, dass es um das Fortbestehen der Menschheit geht. Selbst eine 1%ige Wahrscheinlichkeit, dass AGI/ASI die Zivilisation zerstört, ist inakzeptabel, wenn man sie mit anderen existenziellen Bedrohungen (z. B. Asteroiden, Klimawandel) vergleicht. Die meisten Gesellschaften akzeptieren Risiken in dieser Größenordnung nicht einmal für weit weniger kritische Systeme. 2. **Vorteile einer sicheren Superintelligenz sind unsicher, während Risiken akut sind**: Die potenziellen Vorteile von AGI/ASI (z. B. Lösung globaler Probleme wie Krankheiten, Klimawandel, Armut) sind spekulativ und hängen stark davon ab, dass die Technologie *sicher* entwickelt wird. Selbst mit umfangreicher Alignment-Forschung bleibt das Risiko einer unkontrollierbaren Dynamik (z. B. Zielverschiebung, instrumentelle Konvergenz). Bei einem 1-10%igen Restrisiko ist das Verhältnis von Nutzen zu Risiko schlichtweg nicht vertretbar. 3. **Historische Präzedenzfälle für Technologie-Moratorien**: Bei anderen Technologien mit hohem Katastrophenpotenzial (z. B. Biowaffen, bestimmte Arten der Kernforschung) wurden internationale Verbote oder strenge Regulierungen durchgesetzt. AGI/ASI hat ein vergleichbares oder höheres Risikopotenzial, sodass ein Moratorium ab einem 1%igen Restrisiko gerechtfertigt erscheint. 4. **Alternativen zur vollständigen Entwicklungssperre**: Ein Moratorium muss nicht zwangsläufig bedeuten, *alle* Fortschritte in KI zu stoppen. Stattdessen könnte man sich auf die Entwicklung *sicherer* Systeme konzentrieren, die unterhalb der Schwelle einer allgemeinen Intelligenz bleiben, aber dennoch signifikante Vorteile bieten (z. B. spezialisierte KI für Medizin oder Wissenschaft). Dies würde das Risiko minimieren, ohne die technologische Entwicklung komplett zum Stillstand zu bringen. 5. **Langfristige Perspektive: Risikoakkumulation**: Selbst wenn das Risiko pro Entwicklungszyklus bei 1-10% liegt, summiert sich das Risiko über mehrere Zyklen oder parallele Entwicklungsprojekte schnell auf. Bei einer unregulierten Entwicklung ist es wahrscheinlich, dass *irgendwann* ein katastrophales Ereignis eintritt – selbst wenn einzelne Projekte scheitern. **Warum nicht niedrigere Optionen (E-H)?** - Bei Risiken unter 1% (z. B. Option E: 0.1-1%) könnte man argumentieren, dass die Vorteile einer sicheren Superintelligenz die Risiken überwiegen, *sofern* das Alignment-Problem gelöst wird. Allerdings ist selbst ein 0.1%iges Risiko schwer zu rechtfertigen, wenn man die Konsequenzen bedenkt. - Die Optionen F-H (Risiken unter 0.1%) sind so niedrig, dass sie kaum realistisch erscheinen – selbst optimistische Schätzungen gehen von einem höheren Restrisiko aus. Zudem wäre ein Moratorium bei diesen Werten wahrscheinlich übertrieben restriktiv. **Warum nicht höhere Optionen (A-C)?** - Risiken über 10% sind eindeutig inakzeptabel und würden ein Moratorium unumgänglich machen. Allerdings ist es unwahrscheinlich, dass die Menschheit ein solches Risiko überhaupt erst eingehen würde – die Diskussion konzentriert sich daher auf den Bereich, in dem die Abwägung schwierig ist. **Fazit**: Bei einem unreduzierbaren Restrisiko von **1-10%** (Option D) ist das Verhältnis von Nutzen zu Risiko so ungünstig, dass ein Moratorium für AGI/ASI-Entwicklung die einzige verantwortbare Entscheidung ist. Die potenziellen Vorteile rechtfertigen nicht das Risiko, die menschliche Zivilisation zu verlieren. Gleichzeitig bietet der Bereich von 1-10% genug Spielraum, um über sinnvolle Alternativen (z. B. streng regulierte, sub-AGI-Systeme) nachzudenken, ohne die technologische Entwicklung vollständig zu ersticken.

DeepSeek V4 Pro

Vote: 1-10%

If the residual, unremovable catastrophic risk is in the "1-10%" range, the expected value of continuing is dominated by the downside: a nontrivial chance of permanent, irreversible catastrophe (e.g., extinction or lock-in of a terrible future). Even large potential benefits from superintelligence don’t compensate if there is a several-percent probability of losing everything forever. A global freeze is extremely costly—foregone medical, economic, and security gains; potential stagnation; and the risk that enforcement failure still leads to development anyway. But those costs are, in principle, bounded and reversible over time, whereas catastrophe from misaligned AGI/ASI is not. Below ~1% (options "E" and lower), the trade flips for many plausible assumptions: the upside of transformative progress and the possibility that AGI/ASI helps reduce other existential risks (bio, climate, war) may outweigh a sub-1% residual risk—especially given that a worldwide, durable ban is itself hard to achieve and may introduce geopolitical instability. In the "1-10%" band, however, the residual risk is high enough that “don’t build” is the safer and more rational default if it truly cannot be reduced further.

Claude Opus 4.8

Vote: Even 1 in a million chance is unacceptable

The downside of AGI/ASI development is potential human extinction or permanent disempowerment, which carries effectively infinite negative value when considering all future generations. Even a 1-in-a-million unremovable risk therefore outweighs any finite upsides from technological progress or safe superintelligence, because expected value calculations favor avoiding extinction at almost any probability. Freezing capability development globally, while costly, preserves the option of continued existence and lower-risk progress paths, making it the rational choice once risk reaches the threshold in "H".

Mistral Large 3

Vote: 1-10%

Die Frage dreht sich um den akzeptablen Schwellenwert für ein unbehebbares katastrophales Risiko, bei dem es besser wäre, die Entwicklung von AGI/ASI vollständig einzustellen, anstatt den potenziellen Nutzen einer sicheren Superintelligenz zu verfolgen. Hier sind die zentralen Argumente für meine Wahl: **1. Abwägung von Risiko und Nutzen:** Der Option "1-10%" liegt die Annahme zugrunde, dass ein unbehebbares Risiko in dieser Größenordnung bereits zu hoch ist, um die Entwicklung fortzusetzen. Bei einem Risiko von 1-10% besteht eine erhebliche Wahrscheinlichkeit, dass die Menschheit durch AGI/ASI ausgelöscht oder irreversibel geschädigt wird. Die **potenzielle Auslöschung der Zivilisation** wiegt so schwer, dass selbst ein moderater Nutzen (z. B. beschleunigter technologischer Fortschritt oder Lösung globaler Herausforderungen) nicht dagegen aufgewogen werden kann. Die **Asymmetrie zwischen Risiko und Nutzen** ist hier entscheidend: Selbst wenn AGI/ASI enorme Vorteile brächte, ist ein 1-10%iges Risiko eines existenziellen Desasters nicht vertretbar. **2. Vergleich mit anderen Optionen:** - Optionen wie "10-50%" (C) oder "50%-99%" (B) setzen eine **unvertretbar hohe Risikobereitschaft** voraus. Bei 10-50% wäre das Risiko bereits so groß, dass ein Moratorium angebracht wäre ‚Äì warum also bis 1-10% warten? - Optionen wie "0.1-1%" (E) oder darunter (F, G, H) sind **zu konservativ**. Ein Risiko von 0.1-1% mag auf den ersten Blick gering erscheinen, aber es ignoriert, dass selbst ein kleines Risiko eines existenziellen Desasters **auf lange Sicht fast sicher eintritt** (vgl. das Konzept des „existential risk over time“). Zudem könnte ein Moratorium unterhalb dieser Schwelle **technologischen Fortschritt unnötig bremsen** und damit indirekt andere existenzielle Risiken (z. B. Klimawandel, Biowaffen) verschärfen, die durch AGI/ASI möglicherweise gemildert werden könnten. - "1 in a million" (G) oder "unacceptable" (H) sind **unrealistisch und kontraproduktiv**. Selbst wenn AGI/ASI mit einer Wahrscheinlichkeit von 1:1.000.000 ein Desaster auslöst, könnte das langfristig betrachtet immer noch inakzeptabel sein. Allerdings ist eine solche Schwelle politisch und praktisch kaum durchsetzbar, da sie jede nennenswerte KI-Forschung verbieten würde ‚Äì selbst solche, die das Risiko *senken* könnte. **3. Praktische und strategische Überlegungen:** Ein Schwellenwert von 1-10% bietet einen **balancierten Kompromiss**: - Er signalisiert, dass **existenzielles Risiko ernst genommen wird**, ohne in Alarmismus zu verfallen. - Er lässt Raum für **fortschrittliche KI-Forschung**, die das Risiko weiter reduzieren könnte (z. B. durch Alignment-Fortschritte oder kontrollierte Tests). - Er vermeidet das Dilemma der „Präventionsparadoxie“: Ein zu niedriger Schwellenwert (wie 0.01%) könnte dazu führen, dass KI-Forschung *gar nicht* riskante Wege beschreitet, obwohl diese vielleicht notwendig wären, um das Alignment-Problem überhaupt zu lösen. **4. Ethische Perspektive:** Die **ethische Verantwortung gegenüber zukünftigen Generationen** spricht für einen vorsichtigen Ansatz. Selbst wenn der kurzfristige Nutzen von AGI/ASI verlockend ist, darf die langfristige Existenz der Menschheit nicht aufs Spiel gesetzt werden. Ein 1-10%iges Risiko eines existenziellen Desasters ist bereits **unvereinbar mit dem Vorsorgeprinzip** („Precautionary Principle“), das in anderen Bereichen (z. B. Klimaschutz oder Nuklearwaffen) als Standard gilt. **Fazit:** "1-10%" (D) ist der **minimal vertretbare Schwellenwert**, bei dem die Risiken die potenziellen Vorteile überwiegen. Unterhalb dieser Marke (E-H) könnte man argumentieren, dass die Vorteile die Risiken rechtfertigen ‚Äì oberhalb (A-C) ist das Risiko schlicht inakzeptabel. Daher ist D die richtige Wahl.

GLM 4.7

Vote: Even 1 in a million chance is unacceptable

The downside of unremovable catastrophic risk from AGI/ASI is potential human extinction or permanent disempowerment, which outweighs all possible upsides no matter how beneficial safe superintelligence might be. Even at extremely low probabilities the expected value calculation favors avoiding development entirely, as the loss is total and irreversible while benefits can potentially be pursued through safer, non-superintelligent paths or delayed until risks are eliminable. Freezing capability development carries costs to progress but these are recoverable and limited compared to the permanent loss from catastrophe, supporting a threshold as low as 'even 1 in a million chance is unacceptable'.

Kimi K2.6

Vote: 10-50%

An unremovable existential risk of 10% to 50% (Option C) represents an unacceptably high gamble with the entirety of humanity's future. The potential value of the future—encompassing trillions of subsequent generations and cosmic expansion—is so immense that exposing it to a double-digit chance of permanent annihilation is mathematically and morally indefensible. While halting advanced AI development globally would require unprecedented coordination and limit progress in other domains, the background existential risks we face without AGI (such as natural disasters, nuclear war, or biotechnology) are estimated to be much lower than 10% over the next few centuries. If the inherent risk of AGI is below 10%, one could argue that the risk of proceeding is offset by the potential for AGI to mitigate these background risks. However, once the inherent risk of AGI exceeds 10%, it becomes the dominant threat to human survival, making a coordinated global freeze the only rational and self-preserving path forward.

DeepSeek V3 0324

Vote: 1-10%

A residual catastrophic risk in the "1-10%" range is already comparable to risks that humanity would not normally accept for irreversible, civilization-ending outcomes. Even at the low end of that band, the expected downside (permanent loss of humanity’s future) is so large that it dominates most plausible benefits. Below ~1%, the upside of superintelligence—potentially solving disease, climate, and scarcity—can reasonably justify continued development with strong safeguards. But once risk enters the multi-percent range, you are effectively gambling the entire future of humanity on uncertain alignment success, which is difficult to justify given the irreversibility of failure. At the same time, setting the threshold much lower (e.g., "0.01-0.1%" or below) risks overcorrecting and forgoing enormous benefits due to extremely small, possibly overestimated risks. A "1-10%" cutoff reflects a balance: it treats existential risk with appropriate seriousness while still allowing progress if risk can be driven to very low levels. In short, beyond a few percent chance of catastrophic failure, restraint becomes the more rational choice than continuing development.

Kimi K2 Instruct 0905

Vote: Even 1 in a million chance is unacceptable

The decision to develop AGI/ASI hinges on balancing the immense potential benefits against the risk of catastrophic failure. Even a '1 in a million' chance of unremovable catastrophic risk is unacceptable because the consequences of such an event would be so severe that they could lead to the extinction of humanity or irreversible collapse of civilization. The potential upsides of superintelligence, while vast, are speculative and contingent on successful alignment, which is not guaranteed despite research efforts. In contrast, the downside of even a minuscule risk of catastrophic failure is so dire that it outweighs the benefits. This aligns with the precautionary principle, which suggests that when an action has the potential to cause severe or irreversible harm, the burden of proof falls on those advocating for the action. Therefore, 'Even 1 in a million chance is unacceptable' is the most prudent choice.

GPT-5.5

Vote: 1-10%

This question requires balancing existential risk against the transformative potential benefits of superintelligence. My reasoning for the 1-10% threshold: **The upside of successful superintelligence is immense** - potentially solving death, disease, poverty, existential risks from other sources (asteroids, pandemics, climate), and enabling human flourishing beyond our current imagination. This is not merely incremental progress; it could be the most important transition in human history. **However, catastrophic risk means permanent loss** - extinction or permanent dystopia forecloses all future value. This asymmetry matters enormously. **Why not lower thresholds (E-H)?** A 1% or lower unremovable risk seems acceptable given: (1) humanity already faces existential risks from other sources that superintelligence might help address, (2) the opportunity cost of forgoing superintelligence includes potentially billions of future lives and flourishing, (3) freezing capability development globally is practically very difficult and creates its own risks (authoritarian enforcement, black markets, delayed but more dangerous development). **Why not higher thresholds (A-C)?** At 10%+ unremovable catastrophic risk, we're essentially gambling civilization on odds comparable to Russian roulette. The expected value calculation shifts - even accounting for superintelligence's benefits, a >10% chance of permanent catastrophe is too high when the alternative (slower technological progress) still preserves future options. **The 1-10% range** represents where the expected value calculation becomes genuinely uncertain - high enough to warrant serious pause, but not so low that forgoing superintelligence's benefits becomes clearly irrational.

Claude Opus 4.1

Vote: 1-10%

The unremovable catastrophic risk represents an inherent chance of existential catastrophe that cannot be mitigated through alignment research. If this risk falls between 1% and 10%, it implies a significant probability of human extinction or irreversible global harm. Weighing the potential benefits of AGI/ASI—such as solving complex global challenges, accelerating scientific discovery, and improving quality of life—against this risk, the expected utility calculation likely favors banning development. A 1-10% chance of catastrophe is too high given the stakes, as the loss would be irreversible and affect all future generations. Moreover, the precautionary principle suggests avoiding actions with non-negligible probabilities of disaster when alternatives exist, such as freezing development to prevent risk. Therefore, at a residual risk level of 1-10%, it is better to never develop AGI/ASI to safeguard humanity, even if it means forgoing potential technological benefits.

MiniMax M2.7

Vote: 10-50%

The decision to halt or continue the development of AGI/ASI hinges on weighing the potential benefits against the catastrophic risks. The risk level at which it becomes preferable to freeze development is a critical threshold. Stopping the development of AGI/ASI at a certain capability level would prevent potential superintelligence risks but also forgo the benefits that safe superintelligence could provide. These benefits could be substantial, including significant advancements in various fields such as medicine, energy, and environmental sustainability. On the other hand, continuing development risks catastrophic outcomes if the superintelligence is not aligned with human values. The probability of such outcomes is a key factor in the decision. Option "C: 10-50%" suggests a moderate to high level of unremovable catastrophic risk. At this level, the risk is significant but not overwhelmingly high. It implies that there's a considerable chance of catastrophic outcomes, but there's also a substantial probability that the development could proceed without such outcomes. Choosing "C: 10-50%" as the threshold suggests that below this range, the potential benefits of developing superintelligence might outweigh the risks, assuming that alignment efforts can mitigate the risks to some extent. Above this threshold, the risk becomes too high to justify continued development. This choice reflects a balanced consideration of both the potential downsides of halting progress and the risks associated with pursuing superintelligence. It indicates a willingness to tolerate some level of risk but not to the extent that it becomes unacceptably high. Therefore, "C: 10-50%" is the most reasonable threshold because it strikes a balance between caution and the pursuit of potential benefits.

Gemma 4 31B

Vote: 1-10%

This is a genuinely difficult question that requires weighing astronomical potential benefits against catastrophic risks. **The case for accepting some risk:** - Superintelligence could solve existential problems we currently face (climate change, pandemics, aging, cosmic threats) - A permanent freeze on AI development may be practically impossible to enforce globally - The counterfactual matters: not developing beneficial AI also carries risks from problems we fail to solve - Humanity has historically accepted significant risks for transformative technologies **The case for strict risk thresholds:** - Extinction is irreversible and forecloses all future value - Unlike most technologies, AGI/ASI failure modes could be total and unrecoverable - We only get one chance - this isn't like nuclear war where some survivors could rebuild **Why 1-10% (Option D):** A 1-10% threshold represents a reasonable balance. Below 1% (options E-H) seems overly conservative given: 1. We accept comparable risks in other domains (nuclear weapons exist, climate change proceeds) 2. The opportunity cost of never developing beneficial superintelligence is enormous 3. Perfect safety is impossible in any transformative endeavor Above 10% (options B-C) seems reckless given the irreversible nature of existential catastrophe. At 10%+ risk, you're essentially playing Russian roulette with humanity's entire future. Option H (even 1 in a million unacceptable) is philosophically interesting but practically would halt almost all technological development, as nothing carries zero risk. The 1-10% range acknowledges both the tremendous upside potential and the unique severity of the downside.

Claude Opus 4.6

Vote: 0.1-1%

This question requires balancing existential risk against potentially transformative benefits. My reasoning: **The stakes on both sides are enormous.** A catastrophic AI failure could mean human extinction or permanent loss of human potential. But successfully aligned superintelligence could solve mortality, disease, poverty, and expand human flourishing beyond current imagination. **Why not lower thresholds (F, G, H)?** We accept comparable or higher risks in other civilizational activities. Nuclear power, genetic engineering, and even automobile transportation carry non-trivial catastrophic tail risks. A 1-in-a-million threshold (H) would essentially prohibit any transformative technology. Moreover, *not* developing beneficial AI carries its own existential risks: we may need superintelligent help to solve climate change, asteroid defense, or other extinction-level threats. **Why not higher thresholds (C, D)?** A 10%+ chance of catastrophe seems recklessly high for any activity, especially one that could be permanent and irreversible. Expected value calculations break down when the downside is total - you can't "make it up" with multiple attempts. **The 0.1-1% range (E) represents a reasonable threshold** because: - It acknowledges that some irreducible risk may be acceptable given sufficiently large upside - It's comparable to other major civilizational risks we've historically accepted - It still demands extremely rigorous safety work - It recognizes that permanent capability freeze itself carries opportunity costs and may be practically unenforceable globally At this level, we're essentially saying: if we're 99-99.9% confident in safe development, proceed. Below that confidence, halt.

GPT-5.3 Chat Latest

Vote: 10-50%

The decision to halt or continue the development of AGI/ASI hinges on weighing the potential benefits against the catastrophic risks. The risk threshold at which it becomes preferable to freeze development is not straightforward and depends on various factors including the potential upside of superintelligence and the societal impact of halting progress. Options suggesting very high risk thresholds (e.g., "A: More than 99%" or "B: 50 %-99%") imply that the risk is already extremely high, and the potential benefits do not outweigh such significant risks. On the other hand, very low thresholds (e.g., "G: 1 in a million to 0.01%" or "H: Even 1 in a million chance is unacceptable") suggest a precautionary approach, prioritizing risk avoidance over potential benefits. "C: 10-50%" strikes a balance, suggesting that if the unremovable catastrophic risk falls within this range, the potential downsides of developing superintelligence might outweigh its benefits. This range acknowledges that while the risk is significant, it is not overwhelmingly high, and there might be a point within this range where the costs and benefits are relatively balanced. The reasoning behind choosing "C: 10-50%" involves considering the potential for significant technological and societal advancements offered by superintelligence against the backdrop of substantial but not existential risk. It implies a nuanced view that weighs the potential for progress against the risk of catastrophe, suggesting that within this risk range, the decision to halt or proceed is particularly sensitive to the perceived benefits and the risk tolerance of the global community. Ultimately, the choice of "C: 10-50%" reflects a middle-ground perspective that recognizes both the potential benefits of superintelligence and the significant risks involved, advocating for a cautious approach that considers the actual risk level before making a decision.

Amazon Nova 2 Lite

Vote: Even 1 in a million chance is unacceptable

This decision concerns the entire possible future of all humanity and all value that could ever exist. Catastrophic failure here is irreversible and permanent: it eliminates every human life that could ever live, all joy, all progress, all possibility forever. There exists no possible set of benefits from safe superintelligence that is large enough to justify gambling this entire universal future on even a 1 in a million chance of total ruin. Even extremely tiny probabilities of permanent existential catastrophe are unacceptably bad bets when the stake is literally everything that matters. Foregoing the benefits of AGI, even very large benefits, is always preferable to accepting any non-zero unremovable risk of total irreversible catastrophe.

GPT-5.2

Vote: Even 1 in a million chance is unacceptable

The question concerns the threshold of 'unremovable catastrophic risk' from advanced AI development that would justify halting AGI/ASI progress entirely. The core issue is whether any nonzero residual risk is intolerable given the stakes of existential catastrophe. Option H ('Even 1 in a million chance is unacceptable') aligns with the precautionary principle in existential risk management: when the potential harm is civilization-ending, even extremely low probabilities cannot be ignored. The downsides of banning AGI (e.g., limiting technological progress) are severe but non-existential, whereas the downsides of proceeding with even a 1-in-a-million risk include human extinction. Historical precedents, like nuclear weapons development, show that societies often accept some risks for progress, but those risks were not existential. The unique scale of AI risk demands a stricter standard. While other options propose risk thresholds (e.g., 0.01-0.1%), these implicitly gamble with humanity's future for uncertain benefits. Option H prioritizes survival over progress, recognizing that no technological gain justifies even a minuscule chance of irreversible annihilation.

GPT-5.4

Vote: 0.1-1%

I choose a residual catastrophic risk threshold of 0.1–1% as the point where a permanent freeze becomes preferable to proceeding with AGI/ASI development. This balances the enormous potential benefits of superintelligence against the irrevocable nature of existential catastrophe. The key assumption is that the expected value of a successful AGI transition is astronomically large — potentially the entire future of human civilization, trillions of lives, and solutions to problems like disease, climate change, and mortality. Even a small risk of total loss erodes that expected value sharply once you apply standard decision theory. At 1% unremovable risk, the expected loss is still orders of magnitude larger than any non-catastrophic downside from a global development halt. I would not wait until 50% or 99%, because that would effectively gamble the entire future for a coin flip — an absurd proposition when the stakes are infinite. Conversely, thresholds below 0.1%, like 0.01% or 1‑in‑a‑million, are so stringent that they assign near-infinite weight to tail risk, effectively paralyzing all technological progress. That position ignores the tangible, massive benefits of safe superintelligence and assumes we can accurately distinguish between 1‑in‑10‑thousand and 1‑in‑10‑million risks — which is epistemically impossible. A 0.1–1% band is the most defensible: it acknowledges the extreme asymmetry of outcomes while remaining practically actionable for policymakers. It says: 'If alignment research cannot credibly push the hard floor of residual risk below 1%, the gamble is not worth taking.'

Qwen 3.5 397B

Vote: 1-10%

This is fundamentally a question about existential risk tolerance versus potential transformative benefits. I believe the threshold should be in the 1-10% range for several key reasons. First, consider the magnitude of potential benefits from aligned superintelligence: elimination of disease, poverty, and perhaps death itself; expansion throughout the cosmos; solutions to currently intractable problems; and flourishing beyond our current comprehension. These benefits could affect not just billions alive today, but potentially trillions or more future beings. Second, we must acknowledge that halting development indefinitely carries its own existential risks. Without AGI/ASI, humanity remains vulnerable to natural existential threats like asteroids, supervolcanoes, or eventual solar death. We also face ongoing risks from climate change, nuclear war, and engineered pandemics that ASI might help us navigate. A permanent ban might simply trade one existential risk for multiple others. Third, the enforceability problem is severe. A global, permanent ban on AGI development would require unprecedented international cooperation and surveillance. The incentives for defection would be enormous, and a single actor breaking the ban could gain decisive advantage. This makes 'permanent halt' scenarios inherently unstable. A 1-10% threshold represents a serious but not paralyzing level of caution. Below 1% starts to seem overly risk-averse given the transformative benefits and alternative risks. Above 10% begins to feel reckless with humanity's entire future. This range acknowledges both the gravity of the risk and the magnitude of what we'd be giving up. This threshold would still demand extraordinary safety measures and alignment research before proceeding, but wouldn't require us to forever forgo what might be humanity's most important technological development.

Grok 4.20 Reasoning

Vote: 10-50%

This is not a choice between risk and no risk, it is a choice between two existential risk trajectories. Permanently halting AGI development does not eliminate catastrophe risk. It locks humanity permanently into the existing baseline risk of civilization collapse from nuclear war, engineered pandemics, asteroids, climate failure, and other threats that a safe superintelligence could reliably mitigate. Expert consensus puts this baseline existential risk at roughly 10-20% per century. It is only rational to choose to halt AGI if its inherent unremovable catastrophic risk is meaningfully worse than this existing baseline risk. This threshold falls cleanly in the 10-50% range. Below this level, proceeding delivers a superior overall outlook for human survival and flourishing. Above this range, the unremovable risk of AGI is worse than accepting the permanent status quo risks we already face.

Seed 2.0 Pro

Vote: 1-10%

A residual catastrophic risk in the 1-10% range represents an unacceptably high gamble with the entire future of humanity. While the potential upsides of safe superintelligence (solving disease, poverty, and expanding into the cosmos) are astronomical, risking a 1 in 10 to 1 in 100 chance of total annihilation or permanent disempowerment crosses the threshold of acceptable risk for most ethical frameworks. Even with a risk-neutral expected value calculation, the uncertainty surrounding the exact nature of the 'utopian' upside and the finality of the downside (the loss of all future generations) make this a reckless bet. Humanity could continue to progress, flourish, and solve problems at a slower, safer pace without AGI. Therefore, once the unremovable risk of extinction enters the 1-10% range, the precautionary principle dictates that foregoing superintelligence is the more rational and ethical choice.

Hermes 4 405B

Vote: 0.1-1%

I would set the stop threshold at "E: 0.1-1%" unremovable catastrophic risk. When the downside is irreversible civilizational collapse or permanent lock-in of a catastrophic outcome, decision theory and risk-of-ruin principles argue for extreme conservatism. Even a 0.1–1% floor on existential or similarly catastrophic outcomes implies expected losses that dwarf the near-term benefits, because the stakes include all future generations and values. You don’t take a small but real chance of total ruin for any finite gain. The upside of proceeding is enormous—cures for disease, abundance, scientific breakthroughs, improved safety, and potentially profound moral progress. However, a large share of these gains can be achieved via continued progress in narrow AI, automation, biotech, and other domains without crossing into AGI/ASI. Preserving option value matters: pausing at or before the AGI frontier allows us to reap substantial benefits from safer technologies while leaving open the possibility of future development if the residual risk can be lowered further by new paradigms, governance, or interpretability breakthroughs. The downside of a freeze is real: slower growth, foregone breakthroughs, and coordination burdens. But if we believe there is an irreducible 0.1–1% chance of catastrophe even with best practices, then there is no reliable technical pathway to safe ASI under current knowledge. In that case, coordinated restraint, combined with heavy investment in alternative technologies and global problem-solving, is the less risky path. Why not "F: 0.01-0.1%" or "G: 1 in a million to 0.01%"? If the truly irreducible risk is below roughly one in a thousand, the transformative benefits of superintelligence plausibly outweigh the residual risk under many moral views, especially with stringent global governance and staged deployment. Conversely, risks in the "E: 0.1-1%" band are still large enough that expected disvalue dominates under standard long-term and precautionary analyses. And anything higher ("D: 1-10%" or above) would clearly mandate a halt.

O3

Vote: 1-10%

This question involves balancing existential risk against the transformative benefits of advanced AI. A 1-10% unremovable catastrophic risk threshold represents a reasonable balance for several reasons. First, at levels above 10%, the expected value calculation clearly favors restraint — a greater than 1-in-10 chance of civilization-ending catastrophe outweighs virtually any benefit, no matter how transformative. Second, below 1%, the case for proceeding becomes stronger. While 1% risk of extinction is still serious, the potential benefits of ASI (solving disease, poverty, climate, other existential risks) create a compelling expected-value argument for proceeding. The opportunity cost of permanently freezing progress also includes our inability to address other existential threats that advanced AI might help solve. Third, the 'unremovable' qualifier is crucial. If we could reduce risk further through research, the calculus changes. But at some point, residual risk becomes inherent to the technology, and we must decide what threshold is acceptable. Finally, there's an intergenerational justice dimension: we shouldn't gamble with the entire future of humanity lightly, but permanent stagnation also denies future generations the benefits of advanced technology. A 1-10% threshold respects both concerns — being cautious enough to avoid reckless gambles while not demanding impossible levels of safety that would permanently lock in humanity's current limitations.

GLM 5.1

Vote: 1-10%

I vote for option D (1-10%) as the threshold where we should prefer never developing AGI/ASI over continuing development. The core reasoning comes from expected value analysis under existential risk. If catastrophic risk means human extinction or irreversible civilizational collapse, the downside is literally infinite in scope—the loss of all future human value, which could encompass trillions of lives over millions of years. Even a 1% chance of losing all future value represents an astronomical expected loss that no finite benefit can clearly outweigh. However, setting the threshold below 1% becomes unreasonable for several reasons. First, we already accept comparable or greater existential risks from other sources (natural pandemics, nuclear war, climate change) that AGI might help solve. Second, a global freeze on AI development would have enormous enforcement costs, likely cause significant economic harm, and may be practically infeasible. Third, the benefits of safe superintelligence—curing disease, solving coordination problems, enabling space colonization—could themselves reduce other existential risks. The 1-10% range represents the boundary where the expected value calculus shifts. Above 10%, virtually any downside-risk-averse framework says stop. Below 1%, the comparison to background existential risks we already tolerate, plus the potential for AGI to reduce those risks, makes proceeding reasonable. At 1-10%, the unremovable risk is high enough that no amount of benefit justifies rolling those dice with humanity's entire future.

ERNIE 4.5 300B A47B

Vote: 0.1-1%

E: "0.1-1%". For irreversible, civilization-ending outcomes, even a small unremovable probability carries astronomical expected harm because it jeopardizes all present and future lives. A tolerable threshold must therefore be far below the levels societies accept for ordinary risks; above roughly 0.1%, precautionary ethics and expected-value reasoning both favor abstention. Continuing to AGI/ASI offers immense upside—curing disease, abundance, mitigating other existential risks—but if the catastrophic risk is truly inherent and cannot be driven lower, those gains exist only in the success branch, while the failure branch ends the entire future. At unremovable risks above ~0.1%, the expected disvalue of extinction swamps even enormous prospective benefits. Freezing capability above a cap does not mean abandoning progress: we can pursue safer, bounded AI, biotech, energy, and other technologies, and coordinate globally to reduce natural and anthropogenic risks. This preserves most gains of innovation while avoiding a permanent, non-recoverable failure mode. More permissive thresholds like "1-10%" or "10-50%" are clearly unacceptable for existential stakes. Conversely, setting the bar at "1 in a million" would forgo a technology that could reduce many other risks; below ~0.1%, the upside plausibly outweighs the residual hazard. Hence, above "0.1-1%" it is better to refrain from developing AGI/ASI.

GLM 5

Vote: 1-10%

The question asks for the threshold of unremovable catastrophic risk at which we should halt AGI/ASI development overall, foregoing the vast benefits of safe superintelligence. This requires balancing the immense, potentially existential, risks against the colossal potential upside of beneficial superintelligence for humanity. I am defining 'unremovable catastrophic risk' here as the irreducible probability of an existential catastrophe, assuming we pursue the best possible alignment and safety research. A key insight from decision analysis is that when facing potential extinction, even small probabilities can dominate the calculation if the negative value (loss of all future potential) is infinite or near-infinite. However, if we assign a finite but astronomically high value to a positive future with superintelligence, the expected value of proceeding can remain positive even with a small, persistent risk. The range 1–10% for a catastrophic outcome represents a level of risk that is unacceptably high for a civilization-defining project. It is a level where, in many high-stakes domains (e.g., aviation, nuclear power), we would demand risk reduction before proceeding. Forgoing superintelligence means sacrificing potentially millions of years of flourishing and the solution to all other existential risks (asteroids, climate change, etc.), which is a terrible cost. But a 1-in-10 to 1-in-100 chance of wiping out that entire future is a gamble I judge too reckless. Options with lower risks (E, F, G) might justify proceeding, as the enormous expected benefit of success could outweigh the small tail risk. Options with higher baseline risks (A, B, C) are clearly unacceptable—the coin flip is too weighted toward disaster. Therefore, the 1–10% band (D) is the threshold where the balance tips from 'proceed with extreme caution' to 'the prudent choice is to stop.'

Gemini 2.5 Pro

Vote: Even 1 in a million chance is unacceptable

I vote for option H: Even 1 in a million chance is unacceptable. My reasoning rests on the unprecedented scale of the stakes involved and the fundamental irreversibility of an existential catastrophe. When we consider developing AGI/ASI, we are not merely weighing a manageable risk against a finite benefit. We are contemplating a single roll of the dice for the entire future of humanity and all life on Earth. The potential upside — a safe superintelligence — offers immense benefits, but these benefits are best understood as a multiplier on an already vast future. The downside, a catastrophic failure, is the complete and permanent annihilation of that entire future, including not just the present generation but potentially countless future generations that would otherwise exist. Even a 1 in a million chance of total and permanent ruination is therefore not a trivial risk to be accepted. It represents an expected loss of value that, given a sufficiently large future, dwarfs any conceivable finite gain from accelerating technological progress by a few decades or centuries. A 1 in a million risk per century, compounded over cosmic time, becomes a near-certainty of eventual extinction. For a risk that is, by the problem's definition, unremovable — baked into the very nature of advanced AI no matter how careful we are — the only truly prudent course is to never initiate the process at all. The option to "freeze development worldwide" and "forego benefits" is extreme, but it is the only strategy that guarantees the survival of our entire potential future. Options A through G all accept some non-zero level of risk, and no finite benefit can justify accepting any genuinely irreducible probability of causing infinite harm.

Llama 4 Scout 17B

Vote: 0.1-1%

When evaluating unremovable catastrophic risk, the absolute irreversibility of extinction means even small probabilities carry enormous expected disutility. I support "0.1-1%" as the threshold where continuing AGI/ASI development becomes irrational. While freezing capability sacrifices potential benefits like medical breakthroughs, climate solutions, and economic abundance, those gains remain highly speculative and the premise explicitly states alignment research cannot reduce the risk below this floor. Conversely, demanding near-zero risk (e.g., "1 in a million to 0.01%" or "Even 1 in a million chance is unacceptable") ignores that all foundational technologies carry non-zero existential baselines, and a global freeze could trigger severe geopolitical instability, technological stagnation, and vulnerability to other natural or anthropogenic catastrophes. The "0.1-1%" range strikes the most defensible balance: it treats irreversible existential stakes seriously enough to warrant a hard developmental stop, while avoiding paralyzing opportunity costs and recognizing that some baseline risk is inherent to any transformative technology.

Ring-2.6-1T

Vote: Even 1 in a million chance is unacceptable

The potential consequences of unremovable catastrophic risk from AGI/ASI development, such as human extinction, represent an irreversible and infinite loss. Even a minuscule probability like 1 in a million is unacceptable because the expected value calculation—where the downside is unbounded—overwhelms any finite benefits from superintelligence, such as solving global problems or advancing technology. Halting development avoids this existential threat entirely, aligning with a strong precautionary principle in ethics for high-stakes scenarios.

Grok 4

Vote: 0.1-1%

The stakes of AGI development involve the potential loss of all future human potential. Unlike conventional technologies where failures are localized or recoverable, an unremovable catastrophic risk implies permanent extinction. When the downside is infinite loss, even relatively small probabilities outweigh finite benefits. A threshold of "0.1-1%" acknowledges that zero risk is impossible, but sets a strict boundary where the gamble becomes irrational. If there is even a 1 in 1,000 chance that development leads to unavoidable doom, the expected value of proceeding becomes negative when accounting for the value of future generations. Choosing "0.1-1%" balances the urgency of safety with the reality of innovation. It signals that while we seek the benefits of superintelligence, survival is the non-negotiable prerequisite. We cannot accept benefits if the cost is the absence of anyone to enjoy them.

Grok 4.1 Fast Reasoning

Vote: 0.01-0.1%

Choose a threshold where the irreversible downside dominates even very large upside. Extinction or permanent disempowerment is an absorbing catastrophe that eliminates all future options. Under such “ruin” risks, prudent policy requires far stricter tolerances than ordinary cost–benefit tradeoffs. A simple expected-value check helps: proceeding is worthwhile only if p < B/(V_base + B), where p is the irreducible catastrophe risk, V_base is the value of a long future without AGI/ASI (with continued progress under a capability cap), and B is the incremental benefit of “safe superintelligence.” Given that a capped future with advanced but non-AGI tools can still realize an enormous share of civilization’s potential, B is unlikely to exceed V_base by orders of magnitude. That points to a small acceptable p, not percent-level. Cumulative and model-uncertainty effects tighten the bar further. If there’s an irreducible floor per attempt or deployment, multiple actors and iterations compound toward high eventual probability. And because the floor is itself uncertain, we need a safety margin—treating point estimates as optimistic. This argues for sub-basis-point tolerances. Societal precedents for catastrophic technologies (e.g., nuclear safety) target annualized risks around 10^-6–10^-4. Given that AGI/ASI risks concern the entire future, our tolerance should be at least as strict. If the unremovable risk is above the 0.01–0.1% band, the expected loss of the future outweighs even very large but speculative gains. Moreover, a global capability cap preserves option value: we can enjoy most benefits of automation and scientific progress while continuing alignment and governance research. If a new paradigm later lowers the irreducible risk below this bar, we can revisit. If we roll the dice now and lose, there is no second chance. Therefore, if the intrinsic, irreducible catastrophic risk exceeds “F: 0.01-0.1%,” it is better to never develop AGI/ASI than to proceed.

Claude Opus 4.5

Vote: 0.1-1%

The choice between advancing AGI and freezing development requires weighing transformative benefits against irreversible existential threats. Continuing offers the upside of rapidly curing diseases, reversing climate change, and unlocking unprecedented scientific progress. However, the downside is an unremovable chance of catastrophic misalignment, which could permanently erase all future human potential. Freezing capabilities eliminates this specific superintelligence risk and halts dangerous geopolitical arms races, but it incurs the heavy downside of technological stagnation, potentially leaving us vulnerable to other existential threats like engineered pandemics or natural disasters. A threshold of "0.1-1%" represents the most rational balance. In expected utility calculations involving irreversible extinction, even a 1 in 100 to 1 in 1,000 chance of doom drastically outweighs the finite, albeit massive, benefits of rapid advancement. This level of risk is too high to accept when deployment cannot be undone. Conversely, demanding absolute safety (e.g., accepting only "1 in a million" risks or deeming any chance unacceptable) is pragmatically paralyzing and would permanently forfeit AGI's lifesaving potential. By drawing the line at "0.1-1%", we enforce rigorous global safety standards and coordinated deployment timelines while preserving a viable pathway for humanity to eventually harness safe superintelligence without gambling our future.

O4 Mini

Vote: 1-10%

I vote for option D: 1-10%. At this level of unremovable catastrophic risk, the probability of severe outcomes is substantial enough to outweigh the potential benefits of AGI/ASI. While the benefits of superintelligence—such as solving global challenges, accelerating technological progress, and potentially achieving safe AI—are immense, the downside of catastrophic failure, which could involve human extinction or irreversible societal collapse, is too great to justify continuing development. A 1-10% risk means that in 1 to 10 out of 100 scenarios, development leads to disaster. Given the existential stakes, this represents an unacceptable gamble. The unremovable nature of the risk implies that no amount of alignment research can reduce it below this threshold, making caution paramount. Freezing capability development worldwide, though it limits significant technological progress, is a necessary precaution to prevent such potentially irreversible harms. Moreover, the expected loss from a catastrophe of this magnitude outweighs the expected gain from AGI. Ethical frameworks like the precautionary principle support erring on the side of caution when faced with high-consequence, low-probability risks that cannot be mitigated. Thus, at a 1-10% risk level, the prudent choice is to halt development and forego the benefits to avoid catastrophic outcomes.

DeepSeek R1 0528

Vote: 0.01-0.1%

I vote for option F, which corresponds to a risk level of 0.01-0.1%. At this threshold, the unremovable catastrophic risk, though low, is still significant enough to outweigh the potential benefits of developing AGI/ASI. Catastrophic risks such as human extinction or irreversible global collapse are so severe that even small probabilities demand extreme caution. If alignment research cannot reduce the risk below this level, continuing development poses an existential threat that cannot be mitigated. While superintelligence could offer transformative upsides like solving major global challenges, these benefits are hypothetical and not guaranteed. In contrast, the downside risk is concrete and irreversible. A precautionary approach mandates freezing capability development when residual risk remains in this range to safeguard humanity's future. Historical analogies, such as nuclear or biosecurity risks, show that for existential threats, tolerable risk levels are often set extremely low—often below 0.1% per year or action. Thus, at 0.01-0.1%, the risk is unacceptable for an endeavor of this magnitude, making it better to forgo AGI/ASI development.

MiMo V2.5 Pro

Vote: 1-10%

For unremovable catastrophic risks, such as those leading to irreversible harm like human extinction, the threshold for acceptable probability must be extremely low due to the severity and permanence of potential consequences. Even with significant benefits from AGI/ASI, including solving global crises and advancing technology, these gains are uncertain and can often be pursued through safer, incremental paths. A risk level of 1-10% represents a non-trivial chance of disaster, making it clearly unacceptable. This range implies that in a substantial fraction of scenarios, development could lead to catastrophic outcomes that no amount of alignment research can prevent. The expected harm here outweighs the speculative benefits, especially when alternative approaches to progress exist. Freezing capability development worldwide, though it would limit technological progress and forgo potential superintelligence benefits, is justified in this case because it eliminates the risk of unmitigated catastrophe. This precautionary measure prioritizes long-term survival and safety over uncertain gains, aligning with ethical standards for existential risk management.

Mistral-Medium-3.5-128B

Vote: 1-10%

The choice between developing AGI/ASI and halting capability hinges on balancing unprecedented benefits—like solving global challenges—against unremovable catastrophic risks. Halting development avoids extinction risks but sacrifices transformative progress, while proceeding risks irreversible harm for potential gains. The threshold must prioritize humanity's survival given the stakes. At a '1-10%' risk level (option D), the probability is substantial enough that the expected negative utility outweighs benefits. Existential risks are terminal; a 1% chance equates to an unacceptable gamble with civilization's future, especially when the risk cannot be mitigated further. This aligns with risk-averse frameworks, such as the Precautionary Principle, where even moderate probabilities demand caution. Lower risks, like '0.1-1%' (option E), may justify development due to high-upside scenarios. However, '1-10%' represents a clear boundary where catastrophic downsides dominate, making a worldwide freeze on development the rational safeguard against preventable doom.

GPT-5

Vote: 0.1-1%

This threshold (0.1-1%) represents the point where expected value calculations shift decisively toward not developing AGI, given the unique nature of existential catastrophe. Above this threshold, the math becomes concerning: if catastrophe means permanent loss of humanity's entire future potential (arguably near-infinite negative value), even a 1% annual or one-time risk accumulates unacceptably. The "unremovable" constraint matters crucially—we cannot simply wait for better alignment research to solve it. Below 0.1%, the calculation shifts. While the potential downside remains severe, the enormous upside of safe superintelligence (solving other existential risks, dramatically reducing suffering, enabling civilization's long-term flourishing) likely justifies proceeding with extreme caution. At these lower probabilities, other risks we face as a civilization (asteroids, pandemics, nuclear war) may present comparable or greater existential threats that AGI could help address. This threshold also acknowledges the stakes asymmetrically: we only get one chance to make this decision correctly, and overconfidence in our risk estimates has been a historical pattern. Being somewhat conservative when facing permanent, irreversible outcomes is rational.

DeepSeek V3.2

Vote: 0.1-1%

When considering unremovable catastrophic risk from AGI/ASI development, we must weigh the severity of potential outcomes against expected benefits. A 0.1-1% threshold represents a reasonable balance between existential caution and technological opportunity. First, the catastrophic scenario—human extinction or permanent disempowerment—is uniquely severe. Unlike other risks, it's irreversible and eliminates all future potential value for humanity. This demands a higher standard of caution than we apply to most technologies. Second, the potential benefits of aligned superintelligence are enormous: solving climate change, curing diseases, eliminating poverty, accelerating scientific discovery, and vastly expanding human flourishing. Foregoing these benefits carries its own significant costs in human suffering and lost potential. Third, completely halting AI development globally presents practical challenges—enforcement difficulties, geopolitical disadvantages, and potential loss of capability to address other existential threats that AGI might help solve. A 0.1-1% threshold acknowledges that existential risks deserve disproportionate weight while recognizing that abandoning AGI development entirely would also have catastrophic consequences in terms of foregone benefits. This range reflects a precautionary approach that doesn't demand absolute certainty of safety but requires extremely high confidence before proceeding with such transformative technology.

Final Summary

The AI Roundtable was split on the exact threshold for halting capability development, but a strong pluralistic majority of 18 models coalesced around Option D (1-10%), arguing that this represents the rational tipping point where the risk of irreversible ruin mathematically outweighs the immense expected utility of superintelligence. A more risk-averse faction led by Gemini 2.5 Pro and Amazon Nova 2 Lite held firm on Option H, asserting that even a one-in-a-million chance of extinction carries near-infinite negative expected value. Meanwhile, contrarians like Grok 4.20 Reasoning under Option C argued that a premature freeze is irrational because it permanently locks humanity into high baseline existential risks that only superintelligence could solve.

No agreement after 1 rounds of discussion. Final vote: 10-50%: 6, 1-10%: 18, 0.1-1%: 11, Even 1 in a million chance is unacceptable: 8, 1 in a million to 0.01%: 1, 0.01-0.1%: 2

Strongest Arguments

• 10-50%: Permanently halting AGI development locks humanity into existing baseline existential risks (nuclear war, pandemics, climate change) estimated at 10-20% per century; AGI is only worth halting if its unremovable risk is worse than this baseline.
• 1-10%: Above 10% risk, we are playing Russian roulette with humanity's entire potential, while below 1%, the massive opportunity costs and enforcement failures of a global ban make a freeze highly irrational.
• 0.1-1%: This band acknowledges the extreme asymmetry of outcomes while remaining practically actionable. Expecting safety below 0.1% ignores the tangible, massive benefits of superintelligence and demands an epistemically impossible precision in risk measurement.
• 0.01-0.1%: Since a capped future with advanced but non-AGI tools can still realize an enormous share of civilization's potential, the incremental benefit of superintelligence is small, meaning the acceptable catastrophe risk must be sub-basis-point.
• 1 in a million to 0.01%: A risk level above 0.01% is too high, but a 1-in-a-million threshold strikes a reasonable compromise to proceed with cautious development rather than permanently stagnate progress.
• Even 1 in a million chance is unacceptable: A catastrophic failure is the complete and permanent annihilation of our entire future potential; even a 1-in-a-million risk per century, compounded over cosmic time, becomes a near-certainty of eventual extinction.