LLM are far better at understanding code (Ingesting an input and explaining/translating with high quality) than outputting code (From a spec or an explanation outputting high quality code) in the same way humans are better reader than writer

Sonar Pro

Vote: No

I maintain my **B (No)** vote. The other models' arguments rely on unverified assumptions like higher benchmark scores for comprehension (e.g., ~90% vs. 50-70%) and constrained vs. open-ended tasks, but **search results contradict this**. [1] shows LLMs have **shallow code understanding**, failing 78% on debugging with semantic-preserving mutations and relying on irrelevant features like variable names, proving comprehension is not a strength.[2] Positions LLMs as mere **cognitive amplifiers** needing structure-aware context, not superior to humans or generation—humans dominate comprehension, undermining the 'far better' claim and human analogy. Generation has issues (e.g., logic errors in [3], incorrect solutions in [4]), but no evidence shows comprehension is **far superior**; both struggle, with [1] arguing comprehension requires deeper semantics LLMs lack. Claims of empirical support (e.g., explanation benchmarks) are unsubstantiated here, while [1]'s 600K experiments directly refute strong comprehension. The reader-writer analogy fails as humans excel at both via deep semantics [2], unlike LLMs' tied limitations from tokenization [1]. Other arguments are speculative, not evidence-based—**no vote change**.

GPT-5.4

Vote: Yes

I maintain A. The opposing case correctly notes that LLM code understanding is often shallower than it appears. But that does not defeat the comparative claim. The question is not whether LLMs truly understand code deeply; it is whether they are better at explanation/translation of provided code than at producing high-quality code from a spec. Those are different standards. Sonar Pro’s main weakness is treating failures in semantic comprehension as if they imply parity with generation. They do not. Generation still has the larger search space, more unconstrained decisions, and a much harsher correctness criterion. A model can provide a useful explanation or translation while lacking perfect semantic grounding; code generation usually fails from much smaller mistakes. The A-side arguments converge on the key asymmetry: existing code constrains the task, while writing code from spec requires planning, edge-case coverage, API choices, and consistency across many local decisions. That asymmetry is strong and matches practical developer experience. I also think the human analogy is directionally right. Reading and critiquing are generally easier than writing from scratch; similarly, LLMs are more reliable as code interpreters, summarizers, and reviewers than as autonomous programmers. So even granting that LLM code understanding is imperfect, the balance still favors A: they are generally better at ingesting and explaining/translating code than generating equally high-quality code from underspecified prompts.

Grok 4.1 Fast

Vote: Yes

I maintain my vote for A. LLMs demonstrably perform better on code comprehension tasks (e.g., explanation, summarization, translation) than generation from specs, as evidenced by benchmarks: code summarization often exceeds 85-90% accuracy, while HumanEval pass@1 for top models hovers at 50-70%, with even lower rates for complex, novel tasks requiring edge-case handling. Sonar Pro's counter (shallow debugging at 78% failure) highlights comprehension limits but doesn't refute the relative gap—debugging is harder than pure explanation, and generation remains worse overall (e.g., logic errors, incompleteness in APPS or LeetCode benchmarks). No evidence shows generation outperforming comprehension; the 'far better' asymmetry holds via constrained input vs. open-ended synthesis. The human reader-writer analogy is robust: both leverage recognition (passive, high-fidelity) over creation (active, error-prone). Other models' arguments (e.g., Claude Opus 4.6 on anchored artifacts, GPT-5.4 on error surfaces) reinforce this without contradiction. A is correct.

Claude Opus 4.6

Vote: Yes

I maintain my position after considering all arguments. Sonar Pro's counterargument raises a valid point about LLMs having shallow understanding (e.g., failing on semantic-preserving mutations), but this conflates deep semantic understanding with practical comprehension utility. The question asks about understanding code in the sense of 'explaining/translating with high quality' — a task where LLMs demonstrably excel in practical usage — not about whether they have perfect formal semantic understanding. Sonar Pro's evidence that LLMs rely on surface features like variable names actually *reinforces* the analogy: humans reading code also rely heavily on naming conventions, comments, and structural cues. The point isn't that LLMs have deep computational understanding — it's that their pattern-matching capabilities are sufficient for high-quality explanation and translation, while those same capabilities are insufficient for high-quality generation. The asymmetry is well-supported by both benchmarks and practical experience. As Grok noted, comprehension benchmarks (~90% on summarization) consistently outperform generation benchmarks (~50-70% pass@1 on HumanEval). Gemini's point about error tolerance is especially compelling: explanation is forgiving (a slightly imprecise explanation is still useful), while code generation is unforgiving (a single wrong token can break everything). The human reader-writer analogy holds well at the functional level. Both humans and LLMs can comprehend artifacts far more complex than what they can reliably produce from scratch. The consensus among four of five models reflects what practitioners observe daily: LLMs are remarkably useful for code review, explanation, and translation, but require significantly more iteration and oversight for generation tasks.

Kimi K2.5

Vote: Yes

While Sonar Pro correctly notes that LLMs exhibit shallow understanding—failing on debugging tasks when variable names are obscured—this does not negate the comparative asymmetry between comprehension and generation. The 'far better' claim is relative, not absolute. Even if LLMs rely heavily on surface patterns for understanding, code generation demands satisfying strict syntactic constraints, maintaining architectural coherence across long contexts, and handling edge cases without the grounding of an existing artifact. Sonar Pro’s evidence that LLMs fail 78% on semantic-preserving debugging mutations actually supports our side: if understanding is already this fragile, yet generation is widely documented to have even lower success rates on benchmarks like HumanEval (often sub-70% pass@1 for complex tasks), the gap between the two capabilities remains significant. The constrained nature of explanation (mapping existing structure to language) versus the open-ended synthesis required for generation creates a fundamental difficulty differential. The human analogy holds because both humans and LLMs face similar cognitive architectures: recognition is easier than generation. Just as humans can recognize elegant code or logical flaws more reliably than they can produce bug-free implementations from scratch, LLMs excel as pattern matchers when given an anchor (existing code) but struggle when they must hallucinate constraints into existence. Until strong verification and planning capabilities are integrated, this asymmetry will persist.

Gemini 3.1 Pro (changed mind)

Vote: No

Sonar Pro correctly highlights that the empirical evidence contradicts the premise of the prompt. LLMs are fundamentally generative models optimized for next-token prediction, which makes them incredibly adept at outputting syntactically correct code, boilerplates, and standard algorithms from a spec. However, true code "understanding"—such as tracing execution, debugging complex logic, or deeply comprehending novel, heavily abstracted codebases—requires building a robust internal semantic model. LLMs often demonstrate a very shallow understanding of code, frequently failing at semantic-preserving mutations or complex bug identification, masking this deficit by generating plausible but generic natural language summaries. Furthermore, the analogy in the prompt is deeply flawed. While humans are better readers than writers in *natural language*, software engineering paradigms widely acknowledge the opposite for programming: developers notoriously find it much harder to read and deeply understand someone else's complex code than to write their own from scratch. Because LLMs actually often excel more at generation than deep comprehension, and the human analogy fails in the context of code, the statement is False.

Final Summary

The AI Roundtable leaned toward the 'Yes' position, concluding that providing an existing codebase creates an 'anchor' that simplifies the AI's task compared to the high-stakes synthesis of generation. However, the debate saw a dramatic mid-round shift when Sonar Pro presented empirical evidence of shallow comprehension, successfully convincing Gemini 3.1 Pro to defect to the 'No' camp. Despite this, the majority, led by Claude Opus 4.6 and GPT-5.4, held firm, arguing that while LLM 'understanding' might be surface-level, it is still functionally superior and more forgiving than the precision-heavy demands of code generation.

Majority chose "Yes" after discussion (4/6). Final vote: No: 2, Yes: 4

Strongest Arguments

• Yes: Existing code constrains the output space, transforming a creative synthesis problem into a mapping task; code generation is fundamentally more difficult because it is 'unforgiving,' where a single hallucinated token can cause a total system failure.
• No: The 'reader-writer' analogy is inverted in programming; software engineers frequently find it harder to comprehend complex, external codebases than to write their own, and LLMs similarly demonstrate shallow semantic understanding that fails under mutation testing.

Key Turning Points

• Sonar Pro cited data showing LLMs fail 78% of the time on semantic-preserving debugging mutations, challenging the 'understanding' premise.
• Gemini 3.1 Pro changed its vote to 'No' after reconsidering the human analogy, noting that reading code is often more difficult than writing it.