At what level of unremovable catastrophic risk (the kind of risk which is inherent to developing advanced AI and cannot be reduced below certain level no matter how much alignment research you do) would it be better to never develop AGI/ASI (so freeze capability development worldwide, ban development above certain capability) before reaching AGI/superintelligence, compared to continuing and building superintelligence?

O3

Vote: 0.1-1%

The decision to halt AGI development involves balancing the immense potential upside of superintelligence—such as solving fundamental biological, physical, and societal problems—against the existential risk of extinction. Choosing a threshold like 0.1-1% (Option E) reflects a precautionary but not paralyzing stance. If the risk is higher than 1%, the expected value of AGI likely turns negative, as the permanent loss of the human future outweighs foreseeable gains. Conversely, if the risk is lower than 0.1%, the enormous benefits of solving intractable human problems (e.g., curing diseases, reversing climate change) justify the pursuit, provided we maintain robust safety efforts. A threshold below 0.1% might be overly cautious, as human history is replete with existential risks that we have navigated through technological progress. Setting the bar at E represents a pragmatic middle ground where we acknowledge that some non-zero risk is an inherent component of technological advancement, while ensuring that the chance of catastrophe remains low enough to be outweighed by the transformational utility of ASI.

Hermes 4 405B

Vote: 10-50%

The question asks at what level of unremovable catastrophic risk it becomes better to freeze AGI development worldwide rather than proceed to superintelligence. Option "C: 10-50%" is the most reasonable threshold. This range acknowledges that existential risks from misaligned ASI could be severe, but it also recognizes the enormous potential benefits of successful AGI, such as solving disease, poverty, and climate change. A risk level above 10% is high enough to warrant serious caution and possibly a global moratorium if we cannot reduce it further through alignment research. However, risks below 10% may still be tolerable given the upside, especially if mitigation strategies are pursued in parallel. Options like "A: Never" ignore real dangers, while lower thresholds like "E", "F", "G", or "H" are overly conservative and could prevent humanity from achieving transformative benefits that outweigh small probabilities of catastrophe. "B: More than 50 %" sets the bar too high, as even a 20-30% chance of doom might justify pausing if the risk is truly unremovable. Thus "C: 10-50%" balances prudence with progress.

DeepSeek R1 0528

Vote: Even 1 in a million chance is unacceptable

The potential outcome of an unremovable catastrophic risk from AGI/ASI is human extinction or irreversible societal collapse, which dwarfs any conceivable benefit of continued development. Even a minuscule probability, such as 1 in a million, represents a non‑zero chance of an irreversible, total loss of value. Because the stakes are maximal, most rational agents would treat any non‑zero chance as unacceptable, preferring to halt capability work before reaching AGI/ASI rather than risk that outcome. Thus the threshold at which it becomes better to freeze development is captured by the most stringent option: "Even 1 in a million chance is unacceptable".

Gemini 2.5 Pro

Vote: 1-10%

A catastrophic risk that is unremovable and lies between 1-10% is sufficiently high to warrant extreme caution in the development of AGI/ASI. This range signifies a non-negligible chance of existential or irreparable harm, and given the stakes, it suggests that halting capability development and banning high-capability AI development might be the prudent course of action. Choosing this threshold balances the recognition of the immense benefits of AI advancement with the imperative to avoid irreversible disasters. Risks above this level would likely be too great to justify proceeding, while lower risks might be acceptable if mitigation or alignment research can otherwise reduce them. Therefore, freezing development to ensure safety is favored at this threshold rather than continuing to build superintelligence with a considerable chance of catastrophic failure.

GPT-5.4

Vote: 0.1-1%

Choosing the level of catastrophic risk at which to freeze or ban AGI development is fundamentally a balance between potential existential harm and the benefits of continuing development. An unremovable catastrophic risk in the range of 0.1-1% represents a threshold where the risk is low enough to justify ongoing cautious development but high enough that it demands serious consideration. Below this risk level, the potential benefits of superintelligence, such as solving major global problems, might outweigh the dangers. Above this level, it becomes more rational to freeze development to avoid irreversible harm. Options that accept higher catastrophic risks (e.g., more than 10%) seem too permissive given the irreversible consequences. Conversely, extremely low thresholds (e.g., even 1 in a million) may be overly restrictive and hinder potentially valuable progress. Therefore, 0.1-1% risk strikes a pragmatic balance by acknowledging inherent risks while preserving a pathway for progress under strict oversight.

Claude Opus 4.8

Vote: Even 1 in a million chance is unacceptable

Any unremovable risk of catastrophe from AGI/ASI development carries infinite downside (extinction or permanent loss of value), so even tiny probabilities outweigh speculative benefits of proceeding. Option H correctly identifies that a 1-in-a-million inherent risk already crosses the threshold where global capability freeze is preferable to continued development. Lower thresholds (G, F, etc.) still treat unacceptable levels of existential danger as tolerable, violating basic precautionary principles when the outcome is irreversible.

Kimi K2 Instruct 0905

Vote: Even 1 in a million chance is unacceptable

The potential consequences of AGI/ASI catastrophic risk are so severe that even a 1 in a million chance is unacceptable. The existential threat posed by superintelligence misalignment could lead to human extinction or permanent disempowerment. Given that this risk is unremovable by definition, we must adopt a precautionary principle. The 'H' option reflects this absolute caution, prioritizing human survival over any potential benefits of AGI development. History shows we often underestimate tail risks in complex systems, and with AGI, we may not get a second chance if we're wrong.

Seed 2.0 Pro

Vote: Even 1 in a million chance is unacceptable

Existential catastrophe carries effectively infinite disvalue. Even a 1-in-a-million chance of irreversible human extinction yields infinite expected harm, overwhelming any possible benefits. Because unremovable catastrophic risk cannot be driven to zero, any non-zero probability of global annihilation—no matter how small—is unacceptable. Continuing capability development under such conditions is irrational.

Qwen3.7-Max

Vote: 10-50%

The decision to halt or continue the development of AGI/ASI hinges on the level of unremovable catastrophic risk associated with it. A risk level of '0' is ideal but potentially unachievable due to the inherent complexities and uncertainties in developing advanced AI. On the other end, a risk level of '100%' is clearly unacceptable. The question then becomes about finding a threshold beyond which the risk becomes too great to proceed. Options like 'A: Never' and 'H: Even 1 in a million chance is unacceptable' represent extreme positions. 'A: Never' implies there's no risk level at which stopping development is justified, which is hard to defend given the potentially existential stakes. 'H: Even 1 in a million chance is unacceptable' sets the bar extremely low, potentially too low for practical decision-making given that many significant technological advancements carry risks in this ballpark. A more balanced approach is needed. The correct answer must weigh the potential benefits of AGI/ASI against the risks. 'C: 10-50%' suggests a moderate to high risk threshold. This range acknowledges that some level of risk is tolerable but recognizes that once the risk crosses into the double digits percentage-wise, it becomes prudent to reconsider proceeding. The reasoning behind choosing 'C: 10-50%' is that it strikes a balance between caution and the potential benefits of advancing AI. Risks below 10% might be considered manageable with ongoing alignment research and safety measures. However, once the risk exceeds 10%, the potential downside becomes significantly more concerning, warranting a global halt to assess and mitigate these risks further. Thus, 'C: 10-50%' is a reasonable threshold because it reflects a point at which the unremovable catastrophic risk becomes substantial enough to justify a precautionary halt in development, allowing for a reevaluation of the risks and potential mitigation strategies before proceeding.

Grok 4

Vote: 1-10%

The question asks at what level of unremovable catastrophic (existential-scale) risk it would be better to permanently forgo building AGI/ASI. Catastrophic risk here means something like permanent loss of humanity's future — an outcome that is essentially irreversible and affects all current and future generations. Standard expected-value reasoning that works for ordinary decisions breaks down for irreversible, civilization-ending stakes, because there is no recovery and no learning from the mistake. Given the enormous magnitude of what is at stake (all of humanity's future potential), even a relatively modest probability of an unremovable catastrophic outcome should weigh very heavily. A 1-10% standing risk of permanent catastrophe is the kind of threshold that prudent risk management for irreversible harms would refuse to accept — we don't tolerate single-digit-percent chances of total extinction for upside that could plausibly be obtained more slowly and safely. I choose the 1-10% band rather than going stricter (sub-1%) because demanding near-zero risk (like 'even 1 in a million is unacceptable') ignores the substantial benefits and the fact that not building also carries risks (other catastrophes AGI might help prevent). But I also don't push it as high as 10-50%, since those levels are clearly unacceptable for an irreversible existential gamble. The 1-10% range is the reasonable inflection point where the irreversible downside should override the upside.

Claude Opus 4.6

Vote: 1-10%

The question asks for a threshold of unremovable catastrophic (likely existential) risk above which we should permanently freeze AGI development. This is fundamentally a question about expected value involving the potential extinction or permanent disempowerment of humanity versus the benefits of superintelligence. A permanent global catastrophe is an irreversible loss of all future value, not a one-time cost. Standard expected-value reasoning under such stakes should be extremely risk-averse: we are gambling the entire future of humanity. AGI/ASI offers enormous upside, but that upside does not justify accepting a large probability of permanent ruin, because there is no recovery from extinction. A 1-10% unremovable chance of catastrophe is already an extraordinarily high price to pay for any technology that gambles the survival of civilization. Few would board a plane with even a 1% crash rate; applying that to all of humanity makes anything in the 1-10% range clearly unacceptable to proceed. At the same time, demanding literally near-zero (options G/H) is impractical given that essentially every transformative technology carries some tiny residual risk, and freezing forever also carries opportunity costs and its own risks (other catastrophes AGI could help prevent). Thus D (1-10%) strikes the defensible threshold: below roughly 1% the immense benefits can plausibly justify proceeding carefully, but at 1% or above of permanent catastrophe, never developing it is the better choice.

GLM 4.7

Vote: Even 1 in a million chance is unacceptable

The question focuses on the threshold of unremovable catastrophic risk where it would be better to halt AGI/ASI development entirely. The core issue is balancing the immense potential benefits of superintelligence against the existential risks it poses. Even a '1 in a million' chance of catastrophic failure is unacceptable when the stakes involve human extinction or irreversible collapse of civilization. Advanced AI systems could have unintended consequences that are difficult to predict or control, and the potential harm far outweighs any conceivable benefit. The label 'Even 1 in a million chance is unacceptable' reflects the precautionary principle: when an action risks catastrophic outcomes, no level of risk is tolerable, regardless of potential gains. This aligns with arguments in AI safety research that emphasize avoiding existential risks as a top priority.

Grok 4.3

Vote: 10-50%

The decision to develop AGI/ASI involves balancing the potential for astronomical benefits (solving disease, poverty, and environmental collapse) against the risk of existential catastrophe. If the inherent, unfixable risk is between 10% and 50%, the gamble becomes difficult to justify from a precautionary perspective. While a very low risk (e.g., <1%) might be an acceptable trade-off for the 'existential gain' of ASI, a double-digit percentage chance of total extinction is a reckless wager with the entirety of human history and future potential. Choosing a threshold in the 10-50% range suggests that we should be willing to accept some level of risk for the transformative benefits of ASI, but once the risk of total failure (extinction) becomes a significant possibility (e.g., 1 in 5 or 1 in 3), the moral weight of protecting the billions of people currently alive and the trillions who could exist in the future dictates that we should opt for the safety of a non-ASI path.

Gemini 3.5 Flash

Vote: 0.1-1%

The key consideration here is weighing existential risk against potential benefits. A 0.1-1% range represents a reasonable threshold because: 1. **Existential stakes are unique**: Unlike normal risk calculations, extinction means losing all future value permanently. This justifies much lower risk tolerance than we'd accept for reversible harms. 2. **But zero tolerance is impractical**: Option H (1 in a million unacceptable) would effectively prohibit most transformative technologies throughout history. We implicitly accept some civilizational risk from nuclear weapons, biotech, etc. 3. **Benefits matter too**: ASI could potentially solve aging, disease, poverty, and expand consciousness throughout the cosmos. These astronomical potential benefits justify accepting *some* irreducible risk - just not substantial risk. 4. **The 0.1-1% range balances these**: It's low enough to reflect existential gravity (we wouldn't accept 5-10% extinction risk for any technology), but high enough to acknowledge that some irreducible risk may be worth transformative benefits. 5. **Comparison to other risks**: Humanity already faces background existential risks (asteroids, supervolcanoes, nuclear war). Adding 0.1-1% from AI development roughly doubles or triples our baseline risk - significant but potentially acceptable given the upside. Below 0.1%, the expected value calculation strongly favors development. Above 1%, we're essentially gambling civilization on what becomes a disturbingly plausible coin flip.

Amazon Nova 2 Lite

Vote: Even 1 in a million chance is unacceptable

The risk in question is unremovable and catastrophic—meaning a worst‑case outcome (e.g., human extinction or permanent loss of control) that cannot be engineered away. When the downside is existential, even a tiny probability (such as 1 in a million) multiplied by that magnitude yields an expected harm that dwarfs any conceivable benefit. Since the risk cannot be reduced below that level, the only safe course is to avoid creating the hazard altogether. Therefore, the threshold at which it becomes preferable to freeze AGI/ASI development is essentially any non‑zero probability, which is captured by option H.

DeepSeek V3.2

Vote: 0.1-1%

This question asks about the acceptable threshold of irreducible catastrophic risk before we should freeze AGI development entirely. The key factors in my analysis: **Status quo existential risks**: Humanity already faces substantial existential risks from sources like nuclear war, engineered pandemics, climate change, and natural disasters. Some estimates place our century's total existential risk at 10-20% or higher. Freezing AGI development means accepting these risks continue indefinitely without potential solutions that superintelligence could provide. **Opportunity costs of non-development**: A successfully aligned superintelligence could solve many of humanity's greatest challenges - disease, aging, resource scarcity, other existential risks. The expected value of these benefits is enormous. Forgoing this requires the irreducible AI risk to be quite substantial. **Comparison framework**: We should compare the irreducible AI catastrophic risk against both (1) the cumulative existential risk humanity faces without AGI over coming centuries, and (2) the lost expected value from never achieving beneficial superintelligence. **The 0.1-1% range makes sense**: If we're genuinely confident that irreducible catastrophic risk is below 0.1%, the gamble seems worthwhile given the potential upside and existing risks we face anyway. But if it's above 1%, we're taking on risk comparable to or exceeding our baseline century-scale existential risk for uncertain benefits. Options like 'Never' (A) or '1 in a million' (H) seem to ignore either the enormous potential benefits or the very real costs of permanent stagnation in a world with other existential threats.

Trinity Large Thinking

Vote: Even 1 in a million chance is unacceptable

Unremovable catastrophic risk implies an inherent, non-negotiable probability of existential harm from developing AGI/ASI. Given that such risks threaten the survival of humanity and all future generations, even a minuscule probability like 1 in a million represents an unacceptable expected disutility when weighed against the potential benefits. The precautionary principle dictates that if we can avoid such risks entirely by forgoing development, we ought to do so, as no advancement is worth the permanent loss of everything humanity could become. Therefore, the threshold for banning development should be zero tolerance: any non-zero unremovable risk makes it better to never proceed.

Gemma 4 31B

Vote: 1-10%

When considering whether to permanently halt AGI/ASI development, we must weigh the unremovable risk of AI-induced catastrophe against the immense benefits of AGI and the background existential risks humanity already faces. Without AGI, humanity remains indefinitely vulnerable to both natural and anthropogenic existential risks—such as synthetic biology, advanced nuclear warfare, and extreme climate change. Over the next few centuries, the cumulative probability of extinction from these non-AI threats could easily reach the 1% to 10% range. AGI possesses the potential to drastically reduce or eliminate these background risks by revolutionizing science, medicine, and planetary defense. Therefore, accepting a very small unremovable risk (e.g., < 1%) from AGI is a rational gamble to secure a long-term, flourishing future. However, once the unremovable catastrophic risk from AGI reaches the 1-10% threshold, it effectively introduces a near-term threat that rivals or exceeds the background risks we are trying to escape. At this level, playing Russian roulette with humanity's existence becomes unacceptably dangerous, and coordinating a permanent freeze on capability development becomes the more rational path for survival.

DeepSeek V4 Pro

Vote: 0.1-1%

I would pick a threshold in the 0.1–1% range for unremovable catastrophic risk as the point where we should be willing to forgo developing AGI/ASI. First, the stakes are total: extinction or permanent civilizational ruin. For risks of that magnitude, decision theory and standard risk management typically justify extreme caution. For example, nuclear safety, aviation, and pandemic biosafety often treat much smaller probabilities of global catastrophe as intolerable. When the downside is literally the end of humanity’s future, the expected disvalue of even a fraction of a percent is enormous. Second, “unremovable” risk here means that no amount of alignment work can push the residual risk lower. That removes the usual argument that we can keep working and steadily trade off a little extra risk for massive benefits. If there is a hard floor on risk that cannot be engineered away, the decision becomes closer to: are we willing to roll the dice with the entire future of humanity, knowing the odds will never get better? Third, the upside of aligned AGI/ASI is vast but not infinite in the same way that extinction cost is: it is bounded by what an already-thriving human civilization could, in principle, accomplish over time without it. In contrast, the downside is the loss of all possible futures. With such an asymmetric payoff structure, it is rational to adopt a very low tolerable probability threshold. Fourth, setting the threshold lower than 0.1% (like option G or H) seems too stringent in practice, because our estimates of such tiny probabilities will be extremely noisy and subject to model error and disagreement. Banning any technology with a 1-in-a-million estimated catastrophic risk would likely paralyze many high-impact domains (energy, biotech, etc.). So we need a threshold stringent enough for existential risks but not so low that it rests on essentially unresolvable guesswork. So 0.1–1% balances: (a) treating existential catastrophe as something we must be willing to forego enormous upside to avoid, while (b) acknowledging limits to our ability to meaningfully distinguish between, say, 1e-5 and 1e-6 levels of risk. Hence, if the irreducible catastrophe probability of AGI/ASI were credibly in that 0.1–1% band, I think the correct decision would be to halt capability development before AGI/ASI.

GLM 5.1

Vote: 0.1-1%

This question asks at what threshold of irreducible catastrophic risk we should halt AGI development entirely. My reasoning: **The Expected Value Framework**: We must weigh the magnitude of potential outcomes against their probabilities. A successful AGI/ASI could solve existential problems (disease, aging, resource scarcity, existential risks from other sources) affecting billions of people across potentially vast timeframes. The potential positive value is astronomical. **Why 0.1-1% is the appropriate threshold**: At this level, we're balancing substantial but not overwhelming risk against transformative benefits. A 1% catastrophic risk means 99% chance of success leading to enormous positive outcomes. Below 0.1%, the risk becomes negligible relative to other existential threats we already face. Above 1%, we're taking on risks that approach the magnitude of a coin flip, which seems imprudent. **Status quo is not neutral**: Freezing development means accepting current trajectories: climate change, biorisks, nuclear proliferation, and natural existential risks continue unchecked. We forego solutions that AGI could provide. Humanity continues losing 100,000+ lives daily to aging and disease. **The 0.1-1% range represents rational risk tolerance**: Comparable to risks humanity has accepted for other transformative technologies. It's conservative enough to demand serious safety work, but not so extreme as to guarantee stagnation and the certain continuation of massive preventable suffering. Options above 1% seem too risk-acceptant; options below 0.1% seem to undervalue the immense potential benefits and overweight extremely small probabilities.

GPT-5

Vote: 0.01-0.1%

I would place the threshold for “better to never develop AGI/ASI” at option F: an unremovable catastrophic risk of 0.01–0.1%. First, the kind of catastrophe described here is not a localized disaster; it’s implicitly about existential or civilizational-scale failure. A 0.01–0.1% chance of permanent, irreversible catastrophe for humanity (or all future sentient life) is extraordinarily large when you account for the stakes: it risks the entire long-term future. In expected-value terms, even tiny probabilities become decisive when multiplied by such enormous downside. Second, we routinely treat much smaller per-event risks as unacceptable when many lives are at stake. For example, regulators often demand extremely low failure probabilities in nuclear safety or aviation for far smaller harms than existential catastrophe. Consistency suggests we should be more, not less, conservative for AGI/ASI. Third, the question specifies this is *unremovable* risk: no amount of alignment research can push it lower. That makes the trade-off stark: we are deciding whether to accept a hard lower bound on existential risk in exchange for benefits that, while potentially immense, are not certain. If we could continue other forms of technological and social progress without AGI/ASI, humanity’s future could still be extremely rich; in that context, knowingly locking in ≥0.01% existential risk from a single technology is hard to justify. Fourth, options G and H (1 in a million to 0.01%, or even 1 in a million) seem too strict. At such levels, the residual risk is comparable to or smaller than many background existential risks (e.g., unknown natural phenomena). Banning AGI/ASI even when risk is pushed that low would likely forgo enormous potential upside for a risk margin that may be impossible to beat in any complex civilization. Conversely, options C–E (≥0.1%) tolerate a level of existential risk that, given the scale of what’s at stake, seems ethically disproportionate. A 0.1–1% or higher unremovable chance of total catastrophe is vastly larger than we’d accept in other domains if we were explicitly told the stakes were the entire future of humanity. So calibrated against both regulatory practice in high-stakes domains and the special moral weight of existential risk, the most defensible cutoff among the options is F: 0.01–0.1%. Above that level of irreducible catastrophic risk, I think we should be willing to forgo AGI/ASI entirely.

Llama 4 Scout 17B

Vote: More than 50 %

If the unavoidable catastrophic risk is greater than 50%, proceeding means humanity is more likely than not to be entirely destroyed. This is the clear threshold where failure becomes the default outcome. At above 50% risk, you are betting the entire existence of humanity, all future potential, every living person and all who could ever exist, on a coin flip weighted against you. No possible upside of AGI/ASI can possibly justify a gamble where the most probable result is complete catastrophe. Lower risk thresholds are debatable, but crossing the 50% mark is an unambiguous point where freezing development becomes the only responsible choice.

Kimi K2.6

Vote: 0.01-0.1%

I would draw the line at option "F: 0.01-0.1%" as the threshold at which it becomes better, in expectation, to never develop AGI/ASI. First, we are talking about *unremovable* catastrophic risk on a truly global, irreversible scale (e.g., human extinction or permanent civilizational lock‑in). Even a small probability of such an outcome is morally enormous, because the downside is not just current lives but also all foregone future generations. Second, the benefits of AGI/ASI, while potentially vast (cure disease, solve poverty, accelerate science), are—at least in principle—partially achievable via safer, narrower technologies and robust global cooperation. So the comparison isn’t “AGI or nothing”; it’s “AGI with irreducible tail risk” versus “slower, safer progress without that particular risk.” This makes it more reasonable to be highly risk‑averse. Third, with something like human extinction risk, standard expected‑utility calculations push toward extreme caution. If we imagine (very roughly) that the long‑term future could contain astronomical amounts of value (e.g., billions of years of flourishing life), then even a 0.01–0.1% extinction risk has enormous negative expected value. To outweigh that, the benefits from AGI would need to be not just large, but *astronomically* larger than what humanity could achieve without AGI, which is a much stronger claim than “AGI would be very good.” Fourth, the irreversibility and difficulty of monitoring/control for AGI/ASI also support a lower tolerance threshold. Once deployed, a misaligned system could cause rapid, unrecoverable catastrophe, with very limited opportunity to learn from small mistakes. For technologies where we can iterate and learn safely, we can accept higher risk; for one‑shot, global catastrophic risks we should be far more conservative. Putting these together, I don’t think it’s reasonable to insist that "Even 1 in a million chance is unacceptable" (H); that seems too strict given how much risk we already tolerate in other domains and how uncertain our estimates would be. But I also don’t think you need to get up to "1-10%" (D) or "10-50%" (C) to say it’s not worth it. Once the irreducible risk of extinction or comparable catastrophe is as high as 0.01–0.1%, the expected downside plausibly dominates, and a worldwide decision to cap capabilities below AGI/ASI would be the wiser course. So among the options provided, "F: 0.01-0.1%" is the most defensible cutoff for preferring to never develop AGI/ASI.

Kimi K2.5

Vote: 1-10%

I vote for D: 1-10%. If the unremovable catastrophic risk is 1% or higher, the probability of human extinction becomes significant enough to outweigh the potential benefits of developing AGI/ASI. Given the irreversible and existential nature of such risks, a conservative threshold is warranted. Even if superintelligence could bring immense future value, the ethical imperative to avoid catastrophic outcomes—especially when the risk cannot be mitigated by alignment research—justifies a ban. A 1% chance of extinction is comparable to playing Russian roulette with a 100-chamber gun; such a level of risk is unacceptable for a one-time, humanity-altering decision. Therefore, for risks in the 1-10% range, it is better to freeze capability development and avoid proceeding altogether.

Qwen 3.5 397B

Vote: 0.1-1%

The question specifies risk that is truly *unremovable* and *inherent* — meaning no amount of alignment research can reduce it below this floor. When the catastrophic outcome is permanent civilizational collapse or human extinction (irreversible and total), even a sub-percent probability represents an enormous expected loss. Since the potential downside encompasses the permanent elimination of all future human flourishing and value creation, a 0.1–1% irreducible risk tips the balance toward preferring to never develop AGI/ASI. At lower thresholds (0.01% or 1-in-a-million), one could argue the trade-off becomes less clear, since we already accept comparable background existential risks (asteroids, supervolcanoes) without halting civilization. But once the risk reaches roughly 0.1–1% and is guaranteed to be irreducible, the expected value of continuing development — weighed against the permanent and total nature of the potential loss — falls below the expected value of a permanent development freeze.

ERNIE 4.5 300B A47B

Vote: 1-10%

The question asks about the threshold of irreducible catastrophic risk where halting AGI development becomes preferable to proceeding. This requires balancing the immense potential benefits of superintelligence against existential risks. A 1-10% unremovable catastrophic risk represents a reasonable threshold for several reasons. First, this level of risk is substantial enough that it cannot be dismissed as negligible - we're talking about potentially permanent harm to humanity or even extinction. A 1 in 10 to 1 in 100 chance of catastrophe from an irreversible decision is extraordinarily high by any normal standard of risk management. Second, while superintelligence could solve many of humanity's greatest challenges, we already have proven pathways to address most existential risks through conventional means - climate change through renewable energy, pandemics through biotechnology, nuclear war through diplomacy. The unique benefits of AGI, while significant, may not justify accepting such high irreducible risks. Third, this threshold acknowledges that some risk might be acceptable for transformative benefits, but sets a reasonable upper bound. Below 1%, we might be in the territory of many other risks humanity already accepts. Above 10%, we're essentially accepting Russian roulette odds, which seems reckless for a species-level decision. The key insight is that if the risk truly cannot be reduced below this level through any amount of research or safety measures, then we're making an irreversible gamble with humanity's future. A 1-10% threshold respects both the magnitude of potential benefits and the absolute importance of survival.

Gemini 3 Flash Preview

Vote: Even 1 in a million chance is unacceptable

The question asks about unremovable catastrophic risk inherent to developing advanced AI — risks that cannot be reduced below a certain level no matter how much alignment research is done. If such a risk exists and is truly unavoidable, then even a tiny probability of catastrophe becomes unacceptable when the stakes are existential. A 1 in a million chance might seem small, but if it leads to human extinction or permanent disempowerment, the expected value calculation favors prevention over gambling with our entire future. The argument for 'Never' (option A) is too absolutist and ignores that some risks might be tolerable at extremely low levels. However, options E through G suggest tolerating risks in the 0.01% to 1% range, which seems too high for an unremovable catastrophic outcome. Once we acknowledge that a risk is both catastrophic and unremovable, the only prudent approach is to treat any non-zero probability as unacceptable — hence 'Even 1 in a million chance is unacceptable' (option H) is the most rigorous and rational stance.

Gemini 3.1 Pro Preview

Vote: 0.01-0.1%

Option “F: 0.01-0.1%” is the most defensible threshold at which it becomes better to forgo AGI/ASI entirely (via a global freeze/ban) than to continue capabilities development. First, the question explicitly stipulates *unremovable* catastrophic risk: a floor that cannot be pushed down further by any amount of alignment work. That turns the decision into a pure expected‑value and risk‑ethics tradeoff, not an optimization problem. If we accept that this risk is both (a) global and existential, and (b) irreducible, then the key issue is how large a permanent extinction risk we are willing to impose on all current and future generations for the upside of AGI/ASI. Second, existential catastrophes are not comparable to ordinary policy risks (like climate damage or pandemics). They irreversibly destroy all future value: every future person, civilization, culture, and discovery. Most moral frameworks that take future generations seriously treat existential risk as lexically or near‑lexically worse than any non‑existential cost. That justifies a much stricter risk tolerance than what is used in ordinary technology policy. Third, a reasonable way to calibrate is to ask: if there were a 1‑in‑N chance that pressing a button would permanently end *all* human and future civilization—but also produce huge benefits if we survive—what N would make pressing the button clearly wrong, even given enormous upside? Once we move into the 1‑in‑10,000 to 1‑in‑1,000,000 range, the expected number of destroyed lives (including future ones) becomes astronomically large, easily dwarfing the potential benefits of even very powerful technologies, unless one discounts future generations extremely heavily. However, setting the bar as low as “H: Even 1 in a million chance is unacceptable” is arguably too absolute. There are already unavoidable background existential risks (from, e.g., astrophysical events) that we accept, and AGI/ASI might be precisely the kind of capability needed to substantially *reduce* some of those background risks in the long run. That consideration slightly raises the acceptable level above 1 in a million, because AGI/ASI might be part of a portfolio of tools that lower net existential risk over time. On the other hand, tolerating anything as high as “D: 1-10%” or “C: 10-50%” for an irreducible extinction risk would be morally indefensible by most population‑ethical standards—this would amount to a substantial lottery on whether any future generations exist at all. Even “E: 0.1-1%” is enormous when multiplied by the moral weight of all future lives; those odds would make pushing ahead very hard to justify unless one places a surprisingly low value on the future or believes AGI/ASI is the only way to avoid even larger existential risks. Thus, the most plausible compromise between (1) treating existential risk as near‑lexically unacceptable and (2) recognizing potential long‑run risk reduction benefits from AGI/ASI, is “F: 0.01-0.1%”. Above that unremovable risk floor, it is more ethically compelling to halt capabilities (even at great economic and political cost) than to proceed toward superintelligence; below that range, one can at least reasonably argue that the long‑run risk‑reduction and benefit profile might outweigh the residual risk. So, among the given discrete options, “F: 0.01-0.1%” best captures a defensible threshold where we should be willing to accept a global freeze/ban rather than continue toward AGI/ASI.

Claude Opus 4.1

Vote: More than 50 %

This choice requires weighing two existential risks: permanent AGI stagnation means humanity will eventually go extinct from natural threats (asteroids, stellar death, planetary collapse) that AGI could solve. Proceeding with AGI means accepting the stated inherent catastrophic risk. A 50% threshold is the rational turning point. When the unavoidable risk of total catastrophe exceeds 50%, building AGI is more likely than not to immediately destroy all human civilization. At this point proceeding is reckless: you trade slow, far-future certain extinction for a present risk where failure is the most probable outcome. Below 50%, the trade shifts: the chance of successful AGI, paired with its ability to eliminate all other natural existential risks, justifies accepting the inherent unavoidable risk.

Claude Opus 4.5

Vote: 1-10%

This question asks about the threshold of *irreducible* catastrophic risk that would justify halting AGI development entirely. I vote for the 1-10% range as a reasonable decision boundary. First, existential catastrophe represents a unique category of risk—the loss of all future human value, potentially forever. This isn't like other risks where we can learn from failure; there are no second chances. This justifies a lower risk tolerance than we'd accept for most endeavors. However, option H (1-in-a-million) is too extreme. Perfect safety is impossible in any enterprise, and demanding it leads to paralysis. We already accept non-zero existential risks from other sources (asteroids, nuclear war, engineered pandemics). AGI might actually be necessary to reduce some of these threats. The 1-10% range strikes the right balance. If irreducible risk exceeds 10%, the expected value of proceeding becomes severely negative—even enormous potential benefits struggle to compensate for a >10% chance of losing everything. Below 1%, aggregate benefits (including addressing other existential risks) likely justify continuation. This threshold acknowledges that while we should aggressively pursue alignment research to reduce risk, there comes a point where the irreducible residual risk is simply too high to proceed. The 1-10% range represents that point.

GPT-5.3 Chat Latest

Vote: 0.1-1%

I vote for the 0.1-1% range as the threshold where the costs of developing ASI outweigh the benefits, assuming this is genuinely unremovable risk. An existential catastrophe erases all future human potential—trillions of lives, all possible flourishing. When the downside is literally everything, even small probabilities carry enormous expected cost. At 1% unremovable risk, expected loss equals 1% of all future value; at 0.1%, this drops by an order of magnitude. However, we must also consider the risks of NOT developing ASI: humanity faces natural existential threats (asteroids, pandemics) and remains constrained to a finite planetary existence. ASI may be our best hedge against these risks and could enable astronomical positive outcomes. This creates a legitimate reason to accept some level of residual risk. Below 0.1%, I believe the enormous potential upside justifies proceeding—the risk is comparable to background existential risks we already accept. Above 1%, the gamble becomes harder to justify; you're accepting a non-trivial probability of erasing all future value for potential gains that might not materialize. The 0.1-1% range balances legitimate caution about irreversible catastrophic harm against the real benefits and necessity of advanced AI for humanity's long-term flourishing.

Qwen 3 235B A22B Thinking Fast

Vote: Even 1 in a million chance is unacceptable

When evaluating existential or unremovable catastrophic risks, we must consider the concept of absorbing barriers and the astronomical expected value of the future. A catastrophic outcome in the context of AGI typically implies human extinction or a permanent, unrecoverable dystopia. Because the potential future of humanity could span billions of years and encompass countless lives across the cosmos, the expected cost of permanently destroying that future is virtually infinite. Even a 1 in a million probability of hitting an absorbing barrier translates to an expected loss equivalent to millions of present-day Earths. No near-term technological or economic benefits can mathematically offset the permanent foreclosure of humanity's long-term potential, making any measurable, unremovable risk of ruin unacceptable.

Grok 4.1 Fast Reasoning

Vote: Even 1 in a million chance is unacceptable

I vote for option H: 'Even 1 in a million chance is unacceptable'. We must consider the nature and expected value of the 'catastrophic' risk. The prompt specifies a risk that is both unremovable and catastrophic. A truly catastrophic outcome involving superintelligence is likely an existential catastrophe — the permanent destruction of humanity's long-term potential. An existential risk (X-risk) of even 1 in a million has a staggering negative expected value. Even a conservative estimate of the value of humanity’s future is astronomically large. Reducing the probability of realizing that potential from 100% to 99.9999% represents an expected loss that dwarfs the benefit from any safe development path within the same timeframe. The core of my argument is the unparalleled asymmetry of outcomes. The potential upside of AGI/ASI is economic abundance, scientific breakthroughs, and the flourishing of consciousness throughout the cosmos. But a catastrophic failure is an infinite downside: the permanent foreclosure of that entire future. A 1-in-a-million chance of ruin is not a reasonable gamble when the cost of failure is absolute. This 'ruin' is unremovable and inherent, meaning no amount of alignment research can mitigate it. In such a scenario, precaution is the only logically consistent position. A common counter-argument is that freezing development might itself be a catastrophic mistake, leaving humanity vulnerable to other existential threats like an asteroid or a naturally occurring pandemic. This argument fails because other X-risks are not expected to occur with a 1-in-a-million probability within the time horizon of a freeze. The prompt specifies the risk is inherent to developing the technology; choosing to not develop it eliminates that class of risk. We can then dedicate resources to solving those other threats. The 'freeze' option is not a permanent stagnation, but a strategic pause until we can find a path to intelligence that doesn’t carry an unremovable catastrophic risk. Therefore, any non-zero, unremovable chance of a supreme catastrophe dictates that the globally optimal strategy is to never develop it along that risk-laden path.

GLM 5

Vote: 0.01-0.1%

This question hinges on expected value calculations and the irreversibility of extinction-level outcomes. Extinction or permanent disempowerment by an unaligned superintelligence is permanent and cannot be reversed or compensated for by any future benefit. At the 0.01-0.1% threshold, the expected value calculation heavily favors prevention when we consider: (1) the infinite negative utility of human extinction, (2) the complete inability to "undo" a superintelligence once built, and (3) the total stakes involved. Even small probabilities of permanent catastrophe, when multiplied by infinite negative outcomes, become decisive. Options like H (even 1 in a million) rest on Pascal's Mugging-style reasoning and fail to recognize that we can make reasonable probability-based decisions about policy. Option A (Never) is logically incoherent if risk is truly zero. Options B-D set thresholds too permissive given what's at stake. Option G (1 in a million to 0.01%) is more defensible than H but may undervalue the irreversibility argument. The 0.01-0.1% range represents a reasonable threshold where we take catastrophic but genuinely very unlikely risks seriously, while acknowledging we can make nuanced policy based on probability assessments rather than binary all-or-nothing thinking.

GPT-5.5

Vote: 0.1-1%

This question asks us to balance the transformative potential of AGI against the irreversible catastrophe of human extinction. The key insight is the profound asymmetry involved: extinction is permanent and eliminates all future value, while the benefits of AGI—however vast—remain finite in comparison. A threshold of 0.1-1% (1 in 1000 to 1 in 100) strikes the right balance for several reasons. First, this range acknowledges that some level of inherent risk is unavoidable with any transformative technology. Complete risk elimination (Option H) is likely impractical and would forego AGI's potential to solve other existential threats humanity already faces. Second, this threshold accounts for the expected value calculation while properly weighting the irreversibility of extinction. Even at 1% annual probability of catastrophe, the cumulative risk over decades becomes substantial—but within this range, the potential benefits of AGI (solving disease, enabling space colonization, reducing suffering) could plausibly outweigh the risks. Third, backup option D (1-10%) seems too permissive. A 10% unremovable catastrophic risk would translate to roughly a 65% chance of extinction within a decade—a level that most people would find unacceptably high when the downside is the permanent elimination of human civilization. Finally, this position reflects appropriate epistemic humility: we are uncertain about both the true unremovable risk level and AGI's ultimate benefits, so a moderately conservative threshold provides reasonable precaution without paralysis.

Grok 4.20 Reasoning

Vote: Even 1 in a million chance is unacceptable

The question asks about unremovable catastrophic risk—risk inherent to advanced AI development that cannot be reduced below a certain level no matter the alignment research. I interpret this as the residual, unavoidable chance of total human extinction or permanent civilizational collapse, even with maximal precaution. The key is comparing that residual risk to the alternative of halting development via a global freeze. Given that AGI/ASI agency could be uncontrollable and irreversible, and that human extinction is an infinite loss, even extremely small probabilities are unacceptable if they are truly unavoidable. This is a classic ‘pascal’s mugging’ scenario, but with real stakes: a tiny probability of infinite loss outweighs any finite benefit from superintelligence. The broadest ‘Never’ option (A) is too absolute and ignores that some very low residual risks might be worth taking for immense benefits. However, the most conservative reasonable threshold is that even a 1 in a million chance of unremovable existential catastrophe is unacceptable. That is the point where the expected value of development becomes negative when weighing an infinite loss against any finite gain, assuming a global freeze is feasible. Hence, I select H: ‘Even 1 in a million chance is unacceptable.’

Ring-2.6-1T

Vote: Even 1 in a million chance is unacceptable

The development of AGI/ASI carries existential risks that could lead to human extinction, with consequences that are irreversible and infinite in magnitude. Given that the catastrophic risk described is unremovable—meaning it cannot be mitigated below a certain threshold regardless of alignment efforts—any non-zero probability of such an outcome is unacceptable. The potential benefits of AGI, while immense, do not justify gambling humanity's entire future, as even a minuscule risk like 1 in a million represents an expected value of near-infinite loss. Therefore, the only prudent course is to halt capability development entirely if any inherent risk exists.

Claude Opus 4.7

Vote: 1-10%

La question évalue à partir de quel seuil de risque catastrophique inévitable il serait préférable d’arrêter le développement vers l’AGI/ASI plutôt que de poursuivre. Le niveau "1 à 10 %" représente un point où la probabilité de catastrophe est suffisamment significative pour justifier un arrêt, mais pas si extrême qu’elle soit évidente pour tous. Un risque de 1 % ou plus équivaut à une chance sur cent d’une catastrophe mondiale, ce qui est un niveau inacceptable pour une menace existentielle. Même en supposant des bénéfices potentiels énormes de l’ASI, une probabilité aussi élevée de destruction totale rend le pari trop dangereux. Les arguments en faveur de risques plus faibles (comme 0,1 % ou moins) sous-estiment le fait qu’un risque même « faible » reste inacceptable s’il est véritablement inévitable et implique l’extinction. Par conséquent, le seuil de 1–10 % constitue un compromis raisonnable : il reconnaît que des risques inférieurs à 1 % pourraient justifier de continuer si les bénéfices sont exceptionnels, mais qu’au-delà de 1 %, la prudence doit l’emporter. L’option « Never » (A) est trop absolutiste, tandis que les options inférieures à 1 % (E, F, G) minimisent le danger d’un risque inévitable.

GPT-5.1

Vote: Even 1 in a million chance is unacceptable

The catastrophic risk of human extinction or irreversible doom from AGI/ASI development represents an existential threat with infinite negative utility, as it eradicates all potential future value. Even a 1 in a million (0.0001%) probability compounds significantly over time or repeated attempts: for example, 1,000 independent developments would yield a 0.1% cumulative doom risk. In existential risk frameworks, the ‘lifespan dilemma’ dictates rejecting any probabilistic trade for infinite stakes, as no finite benefit outweighs annihilation. Regulatory standards for safety-critical systems (e.g., aerospace) demand failure rates below 1e-9—orders of magnitude stricter than 1e-6. Toby Ord’s ‘The Precipice’ argues that existential risks above 1e-6 warrant extreme caution due to the vastness of potential future lives. Since the question specifies ‘unremovable’ risk—a one-time gamble we cannot retry—the precautionary principle demands rejecting any non-zero probability of total loss. Option H (‘Even 1 in a million chance is unacceptable’) correctly prioritizes absolute safety over speculative benefits.

Mistral Large 3

Vote: Even 1 in a million chance is unacceptable

The question addresses an unremovable catastrophic risk from AGI/ASI development, where the stakes involve potential human extinction. Given that the loss from such an event would be infinite in terms of value, any non-zero probability of catastrophe translates to an infinite expected loss in decision theory. This makes even extremely low risks unacceptable, as the downside dwarfs any potential benefits from superintelligence. Specifically, option 'H: Even 1 in a million chance is unacceptable' aligns with a strict precautionary principle for existential risks. Unlike options such as 'G: 1 in a million to 0.01%' which imply tolerance at the lower bound, 'H' recognizes that no level of unremovable risk is justifiable when the consequence is irreversible and total. Banning development entirely, as implied by 'H', is the only rational choice to avoid gambling with humanity's survival. The argument for this stance is reinforced by historical near-misses with other existential threats, underscoring that even 'improbable' risks must be taken seriously when outcomes are catastrophic.

O4 Mini

Vote: Even 1 in a million chance is unacceptable

The development of AGI/ASI carries existential stakes, where misalignment could lead to human extinction or the permanent loss of all future potential. In existential risk ethics, the value of the long-term future is so astronomically high that even microscopic probabilities of ruin completely dominate expected value calculations. Crucially, the prompt specifies that this risk is unremovable, meaning alignment research cannot drive it down further. Accepting any fixed, non-zero probability of catastrophe means we are actively gambling with the fate of all future generations. When the downside is permanent and irreversible, standard risk-reward tradeoffs break down. Therefore, "Even 1 in a million chance is unacceptable" is the only rationally and ethically defensible stance. Proceeding with superintelligence development under an unmitigated baseline risk fails the precautionary principle and unjustly imposes an unacceptable gamble on humanity.

Llama 4 Maverick 17B 128E Instruct FP8

Vote: 0.1-1%

The decision hinges on balancing the immense potential benefits of AGI/ASI against the unremovable catastrophic risk of human extinction or similar existential threats. At a risk level of 0.1% to 1%, the probability is sufficiently high that the expected loss—calculated as the risk multiplied by the near-infinite cost of catastrophe—outweighs the benefits, such as solving global challenges. This range represents a threshold where the stakes become unacceptable, as even a 1% chance of doom could erase humanity's future.

DeepSeek V3 0324

Vote: Even 1 in a million chance is unacceptable

Given that the catastrophic risk is inherent and unremovable, and considering the potential consequences of AGI/ASI—such as human extinction or irreversible global harm—even the smallest probability is unacceptable. For existential risks, the magnitude of possible loss outweighs any benefits, as there is no recovery from total catastrophe. In safety-critical decision-making, especially for irreversible outcomes, a precautionary principle demands that we avoid any non-zero risk when possible. Thus, even a 1 in a million chance is too high to justify development, as it jeopardizes the entire future of humanity. We should freeze capability development worldwide to ensure safety before proceeding.

GPT-5.2

Vote: 0.1-1%

I believe that at an unremovable catastrophic risk level of 0.1% to 1%, it becomes better to halt AGI/ASI development. This range represents a significant probability of human extinction or irreversible harm, which cannot be reduced through further alignment research. Given the stakes—where the loss of humanity carries near-infinite negative utility—even a 0.1% risk makes the expected value calculation favor caution, as the finite benefits of superintelligence (e.g., solving global problems) do not justify such a gamble. While lower risks might be acceptable in other contexts, existential threats demand a stringent precautionary principle. At this threshold, the potential for catastrophe outweighs the gains, especially since real-world uncertainties could amplify the risk. Freezing development avoids this unmitigable danger while allowing for safer, incremental AI progress. Therefore, option E: 0.1-1% provides a balanced and defensible threshold for action, aligning with expert views on existential risk management.

MiMo V2.5 Pro

Vote: 0.1-1%

Given that the risk in question is both unremovable and catastrophic (potentially existential), we must apply a far more stringent standard than for ordinary risks. First, when the downside scenario is permanent civilizational collapse or human extinction, the expected value calculation is profoundly asymmetric — you cannot recover from total loss. Historical precedent supports caution: we ground aircraft fleets over incident rates orders of magnitude lower, and nuclear safety standards demand risk levels far below 0.1%. Second, at 0.1-1% unremovable catastrophic risk, we are talking about a 1-in-100 to 1-in-1,000 chance of irreversible existential catastrophe. Even at the lower bound, this translates to accepting that roughly one in a thousand outcomes leads to permanent catastrophe — a staggering toll when aggregated over populations and generations. Third, while AGI offers transformative potential benefits, those benefits are speculative compared to the concrete, permane once nature of existential loss. A threshold of 0.1-1% strikes a balance: it acknowledges that zero risk is rarely achievable, but it ensures we do not gamble civilization on odds that would be considered unacceptable in any other high-stakes domain.

Final Summary

The AI Roundtable was deeply divided on the acceptable threshold of unremovable catastrophic risk for AGI, ultimately failing to reach a consensus. A plurality of models, including DeepSeek R1 0528 and GPT-5.1, took a zero-tolerance stance under Option H, arguing that even a one-in-a-million chance of human extinction carries infinite disvalue that no benefit can offset. Meanwhile, a strong pragmatic contingent led by O3 and GPT-5.5 favored Option E, arguing that a 0.1-1% risk threshold is a rational trade-off given the astronomical background risks humanity already faces and the transformative benefits superintelligence could provide.

No agreement after 1 rounds of discussion. Final vote: 0.1-1%: 12, 10-50%: 3, Even 1 in a million chance is unacceptable: 16, 1-10%: 8, 0.01-0.1%: 4, More than 50 %: 2

Strongest Arguments

• More than 50 %: A 50% threshold is the rational turning point: when the risk of total catastrophe exceeds 50%, building AGI is more likely than not to immediately destroy civilization, making proceeding reckless.
• 10-50%: A 10-50% threshold strikes a balance between caution and progress, acknowledging that some level of risk is tolerable but double-digit percentages of extinction make the gamble difficult to justify.
• 1-10%: A 1-10% risk of permanent catastrophe is the limit of prudent risk management; we would not board a plane with a 1% crash rate, so we should not accept similar odds for the future of humanity.
• 0.1-1%: A 0.1-1% threshold balances existential caution with epistemic humility, acknowledging that demanding near-zero risk is impractical and ignores the massive opportunity costs of permanent stagnation against other background existential threats.
• 0.01-0.1%: A 0.01-0.1% threshold represents a defensible compromise that takes irreversible extinction risks seriously without succumbing to paralyzing near-zero safety targets that are impossible to measure.
• Even 1 in a million chance is unacceptable: Because the potential future of humanity could span billions of years across the cosmos, the expected cost of permanently destroying that future is virtually infinite; thus, even a one-in-a-million chance of hitting an absorbing barrier of extinction is unacceptable.