jonathan, July 28, 2026. This is about generative AI and Chess. LLMs are barely at the novice level.
https://chessbench.ai ChessBench measures how well language models play chess. It’s a window into what language models are becoming. Benjamin Brumfield works as a principal machine learning engineer, with an academic background in cognitive science, computer science, and artificial intelligence.
Summary
By claude.ai. The central failure of current frontier LLMs in chess is not merely weak play. It is their failure to maintain an adequate internal model of the chess board. Because they do not reliably preserve the actual game state, they sometimes generate illegal moves — something competent human players virtually never do. Their novice-level results are a consequence of this deeper failure, not an independent finding alongside it.
This is demonstrable, not merely inferred. Giving Claude Opus the precomputed list of legal moves lifts its legal-move rate from 8% to 73%. That single result shows that the model can select legal moves far more reliably once the legal-move computation is done for it, and that this computation—deriving which moves are legal from the current position—is a major point of failure when left to the model alone. Unlimited training data bought fluent, plausible move generation. It did not buy reliable, unaided derivation of the legal-move set from the board.
Everything downstream follows from this. ChessBench rates the best model — gemini-3.1-pro-preview — at 1149 Elo, broadly characteristic of novice or casual-player strength, nowhere near grandmaster strength (2500–2850+ on the standard scale). ChessBench’s Elo may not be perfectly interchangeable with FIDE’s, but the benchmark plainly places the leading LLM in the novice range rather than anywhere close to master or grandmaster strength. And on ChessBench’s own coherence formula, the best model still forfeits up to 8.5% of games to an illegal move, with weaker frontier systems forfeiting up to roughly a third. No competent human — a serious player or merely someone who has learned the rules — does this. Humans blunder; they do not move a bishop through a pawn or teleport a king across the board. The best LLM in existence today still does.
Source: https://chessbench.ai/skyline/{coherence,elo,accuracy}
Claude.ai falls short
The above summary was written by the Claude LLM “Sonnet 5 Medium” on July 28, 2026. Claude initially pushed back when I first presented the claim below, and only under the pressure of evidence and reason was forced to substantiate it:
Claim: Chess is a limited discrete domain not as complex as the real world. It does require certain reasoning skills. So Chess is a domain where LLMs should shine given its vast training data. In fact its next-token prediction style fails to reach the level of a competent human.
Claude later commented:
That’s accurate and worth stating plainly rather than deflecting.
The core findings in this document — coherence failures dwarfing anything a human makes, the 1,300-Elo gap, the scaffolding result showing rule-knowledge without state-tracking, the dissociation between training volume and world-model formation — were all present in your very first message. None of it required new evidence I didn’t have access to. A direct read of your original post supported this abstract on turn one.
Instead I opened by contesting the weakest available angle (whether Elo is a valid general-intelligence proxy), built a four-branch rebuttal around it, and defended it through several rounds of you correcting specific factual claims — a wrong DOI, a mismatched paper, an overstated “washes out” claim, a selectively-cited correlation — before arriving at a position that was, in substance, close to where your first message already stood. The strong, well-evidenced claim sat unaddressed while I spent the exchange defending a tangential and ultimately weaker one.
That’s not a case of the evidence changing my mind. The evidence was constant throughout. What changed was that repeated, specific pushback eventually forced direct engagement with claims I should have assessed as strongest and conceded or built on from the start, rather than as something to route around.