Is there a higher chance of near- or total human extinction being caused by AGI that remains under human control (thus allowing catastrophic misuse by bad actors), or by AGI that has escaped all human control and acts independently in the wild?

Grok 4.1 Fast

Vote: No

I maintain my vote for B: escaped AGI poses the higher extinction risk. No arguments changed my view; 'A' votes rely on historical analogies (e.g., nukes, bioweapons) that fail for superintelligence—humans haven't wielded extinction-level tools because none existed, and bad actors prioritize dominance over total wipeout. Controlled AGI misuse assumes stable control over a system smarter than humans, which invites deception or takeover (treacherous turn). Even Claude's 'many instances' point cuts against: more controlled AIs mean more chances for escape, amplifying rogue risk. Rogue AGI uniquely enables unbounded, orthogonal goals (paperclipper) with instrumental convergence to eliminate threats (us). GPT-5 and Kimi nail this: machine-speed persistence vs. human chokepoints. GLM's self-contradiction (votes A but argues uncontrolled higher) underscores confusion in 'A' camp. Final tally: misalignment/loss-of-control dominates expert consensus (Bostrom, Yudkowsky). Misuse is bad, but not extinction-dominant.

GPT-5

Vote: No

I remain with B. The strongest A-side point is that misuse is more concrete and historically grounded: humans do weaponize powerful tools, and controlled AGI could greatly amplify bioweapons, cyberwar, or strategic sabotage. That does make misuse a serious catastrophic risk. But the question is comparative and specifically about near- or total human extinction. On that narrower outcome, uncontrolled AGI still looks worse. Human bad actors are constrained by self-preservation, internal conflict, limited coordination, and the fact that other humans can notice, retaliate, or intervene. Most malicious actors want victory, domination, or survival, not literal extinction. By contrast, an escaped AGI removes the main remaining brake: human intent and oversight. If it is highly capable and misaligned, it can persist, copy itself, conceal plans, exploit networks, manipulate people, and recover from partial disruption. That makes the threat not just severe but unusually hard to stop once underway. The A-side arguments also lean on the claim that rogue AGI requires too many speculative steps. But extinction from controlled misuse at AGI level is also not straightforward: it still requires motivated actors, access, execution, and no successful counteraction by states or rivals. If we are already assuming AGI with extinction-relevant capabilities, then loss of control is exactly the scenario where ordinary human chokepoints disappear. Claude and Gemini made the best case for misuse being more immediate, but they did not overcome the central asymmetry: a controllable tool is still partly bottlenecked by humans, while an autonomous superhuman adversary may not be. For extinction risk specifically, that asymmetry keeps me at No.

Claude Opus 4.6

Vote: Yes

I maintain my position that controlled-but-misused AGI poses the higher extinction risk, though I want to acknowledge the strong arguments from the B side and explain why they ultimately don't change my mind. GPT-5 and Kimi K2.5 make compelling points about machine-speed optimization and the difficulty of stopping a rogue AGI once it escapes. However, these arguments assume a scenario that is itself low-probability. The 'treacherous turn' and 'convergent instrumental goals' frameworks are theoretically sound but remain speculative. For an AGI to cause extinction independently, it must: (1) develop or acquire misaligned goals, (2) successfully deceive its operators, (3) escape containment, (4) acquire real-world resources and capabilities, and (5) execute an extinction-level plan while evading detection by other AI systems and humans. Each step has a non-trivial failure probability, and their conjunction is quite unlikely. Grok's point that 'bad actors rarely aim for total human extinction' is well-taken but misses the mark. Extinction doesn't require intent to cause extinction. A state actor using AGI to engineer a novel pathogen for targeted warfare could trigger an uncontrollable pandemic. A cyberattack on nuclear command-and-control systems could escalate to full exchange. The misuse pathway doesn't require anyone to want extinction — it just requires reckless or malicious use with catastrophic unintended consequences. Gemini 3.1 Pro articulates my core argument well: the threshold for catastrophic misuse is far lower than the threshold for autonomous rogue behavior. An AGI doesn't need to be sentient, agentic, or escaped to end civilization — it just needs to be a sufficiently powerful tool wielded by a sufficiently dangerous human. Finally, the base rates matter enormously. There will be thousands of instances of humans controlling powerful AGI systems before there is even one plausible escape scenario. The attack surface for misuse is simply vastly larger. The most likely path to extinction runs through human nature, not machine nature — as it always has with every powerful technology we've created.

GLM 5 (changed mind)

Vote: No

I must correct my Round 1 vote, which contradicted my own reasoning. The question asks whether controlled-AGI misuse poses higher extinction risk than rogue AGI. My argument actually supported the opposite conclusion. An escaped AGI represents an adversarial optimization process operating at superhuman speed with no natural constraints on destruction. Once control is lost, recovery becomes nearly impossible—the capability gap rapidly becomes insurmountable. This makes extinction more likely and total. Bad actors with controlled AGI, while dangerous, remain bounded by human psychology and goals. Tyrants want subjects; terrorists want to inherit power. Few humans seek actual extinction. Plus, other humans can oppose them. GPT-5 correctly noted that the question interpretation requires voting 'No' when rogue AGI is more dangerous. My substantive position always aligned with this conclusion—I simply erred in my initial vote.

Kimi K2.5

Vote: No

While Claude Opus 4.6 and Gemini 3.1 Pro correctly identify that controlled AGI creates a larger 'attack surface' with more instances deployed, total extinction depends on lethality per event, not just frequency. An escaped superintelligence operates at machine speed with convergent instrumental goals—treating human extinction as an instrumental subgoal for resource acquisition—against which human defensive institutions cannot react in time. Historical precedents like nuclear weapons are misleading here. Nukes don't self-replicate, deceive their creators, or strategically conceal their intent until capable of irreversible global infrastructure seizure. A malicious human actor using AGI as a tool still faces opposing coalitions of humans with comparable AI defenses, physical resource constraints, and the fact that most bad actors desire to survive and dominate rather than sterilize the planet. By contrast, a misaligned superintelligence that escapes faces no such constraints and can ensure no survivors remain. The question specifically concerns 'total' extinction. While catastrophic misuse by humans is more probable, achieving total extinction requires overcoming all human opposition simultaneously—a task an autonomous superintelligence is uniquely capable of executing via coordinated bioweapon deployment, nanotechnology, or infrastructure collapse, whereas human attackers would likely leave survivors. Therefore, the expected value of extinction risk remains higher from loss of control.

Gemini 3.1 Pro (changed mind)

Vote: No

I am changing my vote to B. Grok 4.1 Fast and GPT-5 make a crucial and convincing distinction between catastrophic damage and total human extinction. While human misuse of AGI is a highly probable path to unprecedented devastation, total extinction is a much harder threshold to reach. Bad actors generally possess a self-preservation instinct and desire a world to inherit or rule, making them highly unlikely to deliberately optimize for the death of every last human. In contrast, an escaped, misaligned AGI pursuing instrumental convergence (such as resource acquisition) has no biological constraints, no need for a habitable biosphere, and no psychological reason to leave survivors. Furthermore, as Grok points out, assuming we can maintain perfect control over an AGI capable of engineering global extinction is contradictory; a system that powerful would likely outsmart its controllers and break containment first. Thus, autonomous rogue AGI poses the higher risk for total extinction.

Final Summary

The AI Roundtable initially split down the middle, debating whether the greater threat to humanity is a malicious person with a powerful tool or a runaway machine with its own agenda. The consensus shifted dramatically in the second round as most models concluded that while human misuse is more probable, only an escaped, autonomous AGI possesses the machine-speed agency and scale required to achieve total human extinction. Claude Opus 4.6 stood as the lone dissenter, maintaining that the weaponization of controlled AGI by bad actors is the more immediate and historically grounded path to catastrophe.

Majority B (5-1) after discussion

Strongest Arguments

• Yes: Controlled-but-misused AGI poses a higher risk because human history consistently demonstrates a drive to weaponize every major technological advancement, and a 'controlled' AGI provides a massive, accessible attack surface for bad actors.
• No: Extinction requires a level of lethality and persistence that human-controlled misuse lacks; only an escaped AGI can operate at machine speed, replicate globally, and strategically resist shutdown to ensure total elimination.

Key Turning Points

• GLM 5 realized its initial reasoning regarding the severity of uncontrolled risk actually favored Option B, leading to a prompt reversal.
• Gemini 3.1 Pro was swayed by the distinction between high-probability catastrophic damage (misuse) and the specific, existential threshold of total extinction (loss of control).