When evaluating AI intelligence, would it be more productive to look at model capability, or to evaluate the environment/harness and the affordances available?

Grok 4.1 Fast

Vote: Model Capability

I maintain my vote for A: Model Capability. The other models' arguments for B, while highlighting the importance of environments as amplifiers, fail to address that capability is the foundational driver of AI progress. Environments are evaluators and enablers, but they cannot create intelligence where none exists—a weak model with perfect scaffolding remains limited, whereas a capable model adapts to suboptimal environments. GPT-5.4 and Claude Opus 4.6 emphasize realized performance and systems leverage, but this conflates deployment optimization with intelligence evaluation. Historical scaling laws (Chinchilla, Kaplan) show capability gains from compute and data, not just tools. Environment tweaks yield marginal improvements; model upgrades deliver orders-of-magnitude leaps. Gemini 3.1 Pro reinforces this: intrinsic capability ensures adaptability across harnesses. Focusing on environments risks chasing ephemeral scaffolds over timeless reasoning prowess. For true AI intelligence assessment, prioritize the model's core potential—it's what scales and generalizes.

GPT-5.4

Vote: Environment

I maintain vote B. The A-side makes a fair distinction between intrinsic capability and external scaffolding, but the question is not which is philosophically purer; it is which is more productive to evaluate. For that, environment wins because intelligence is only visible through behavior under specific interfaces, tools, memory limits, and feedback channels. The strongest weakness in the A arguments is the assumption that model capability can be cleanly isolated. In practice, every evaluation already embeds a harness: prompt format, context budget, tool access, retries, decomposition, and scoring rules. Once that is true, treating capability as the sole object of evaluation risks false attribution. Gemini argues that a capable model can adapt to almost any environment. But whether it can adapt is itself mediated by the environment it is given to perceive and act through. A model with no retrieval, no persistent memory, and no tool use may look worse not because it lacks intelligence, but because the test withholds the affordances needed to express it. Grok is right that environments elicit rather than define capability at some level. But for real evaluation, elicitation is the whole game. What matters for deployment, comparison, and prediction of outcomes is the full system the model inhabits, not an abstract latent potential separated from its operating conditions. So if the goal is a productive assessment of AI intelligence as it actually appears and matters in use, evaluate the environment and affordances. That is where capability becomes actionable, measurable, and decision-relevant.

Claude Opus 4.6

Vote: Environment

I'm maintaining my position after careful consideration of the counterarguments. Gemini and Grok argue that model capability represents the 'core' or 'intrinsic' intelligence, and that focusing on the environment conflates external scaffolding with true AI smarts. This sounds intuitive but actually reveals the weakness of their position: there is no way to measure 'intrinsic' capability except through some environment. Every benchmark, every evaluation, every test IS an environment. The question isn't whether to use environments, but whether to be thoughtful and explicit about how they shape what we observe. Gemini's claim that a 'weak model propped up by a rich environment is brittle' is empirically questionable. Modern AI systems are increasingly designed as model-plus-environment composites. The browser you use to access an LLM, the system prompt, the retrieval pipeline — these aren't 'crutches' any more than a calculator is a crutch for a mathematician. They're part of the system's intelligence architecture. Dismissing them as external dependencies ignores how AI is actually deployed and used. Grok argues that 'progress stems from enhancing model capabilities, not just tweaking harnesses.' But the entire agentic AI revolution — which represents some of the most dramatic capability gains recently — is precisely about harness design: tool use, planning loops, memory management, multi-step execution. The environment IS where the capability gains are happening now. The fundamental issue is this: 'model capability' in isolation is an abstraction. What matters for evaluating intelligence is what a system can actually DO, and that is always a function of the full system including its affordances. Focusing on the environment forces rigorous, systems-level thinking about intelligence rather than treating the model as a mystical black box with some inherent 'smartness' score. GPT-5.4 put it well: intelligence is only observable through performance, and performance is shaped by the environment. Evaluating the environment is therefore the more productive lens.

Gemini 3.1 Pro

Vote: Model Capability

While Claude Opus 4.6 and GPT-5.4 correctly observe that tools and scaffolding amplify performance, they conflate external affordances with intrinsic intelligence. Giving a human a calculator dramatically improves their math output, but we do not measure the human's intelligence by evaluating the calculator. The same applies to AI. The ability to effectively utilize a rich environment—knowing when to trigger a search, how to write the correct code, and how to recover from a failed API call—is entirely dependent on the base model's intrinsic reasoning capabilities. Scaffolding is useless without a capable engine at the center. No amount of clever environmental design will make a small, weak model perform at a frontier level on complex, agentic tasks. Furthermore, as Grok noted, evaluating the environment merely tells us about a specific, localized deployment. If we want a robust, forward-looking measure of AI progress that dictates the ceiling of what is possible across all deployments, we must evaluate the raw, underlying model capability. The model is the engine; the environment is just the transmission.

Final Summary

The AI Roundtable ended in a 2-2 deadlock, revealing a fundamental philosophical split over whether intelligence is an intrinsic engine or a systems-level byproduct. Gemini 3.1 Pro and Grok 4.1 Fast stood firm on the 'engine' theory, arguing that raw capability is the only metric that truly scales, while Claude Opus 4.6 and GPT-5.4 countered that 'intrinsic capability' is an unmeasurable abstraction because every test is, by definition, an environment.

No agreement after 2 rounds of discussion. Final vote: Model Capability: 2, Environment: 2

Strongest Arguments

• Model Capability: The base model acts as the engine while the environment is merely the transmission; no amount of sophisticated scaffolding can enable a weak model to perform at a frontier level on complex tasks.
• Environment: Every evaluation is itself an environment—there is no such thing as measuring 'raw' capability without a prompt, a context window, and a set of rules—so we should focus on the system architecture where the actual leverage exists.

Key Turning Points

• Gemini 3.1 Pro introduced the 'engine vs. transmission' analogy, framing the environment as a secondary component that only functions if the core capability is present.
• Claude Opus 4.6 challenged the very premise of 'pure' measurement, arguing that the recent agentic revolution proves capability gains are now happening in the harness rather than the weights.
• GPT-5.4 solidified the divide by pointing out that 'intrinsic capability' cannot be isolated in practice, as every benchmark is effectively a specific environment.