Why I still score my own benchmark, even though two LLM judges do it for me

Last time I wrote about my benchmark (the code that kills your browser), I skipped the boring part. Sixteen models, six attempts each, one task – that is 96 HTML files somebody has to open, look at, click around and score on five criteria. That somebody was me. With coffee, over a few evenings. By now the benchmark holds 381 attempts.

Then I added a fourth task and there was so much of it that something had to give.

So I added a judge. Two, actually.

The judge doesn’t get a question, it gets an evidence folder

The simplest version of “LLM as a judge” goes like this: you paste in the code, you ask “is this good, score it 1-10”, you get a seven. Always a seven… :P

That doesn’t work, because the task is to produce a working page, not pretty code. Code can look immaculate and blow up in the console within the first second. It can be ugly as sin and run just fine.

So instead of asking, I build an evidence folder. Underneath sits Playwright, which runs the artifact headless and collects:

• shot-1.png, shot-2.png, shot-3.png – the same 1280×800 viewport at 1, 6 and 12 seconds after load,
• shot-full.png – the whole page, top to bottom, taken after scrolling through it,
• interact-1.png … interact-3.png – the page after one interaction each, every shot on a freshly loaded page,
• console-errors.json – everything that came in as pageerror and console.error,
• interactions.json – a plain-words description of what each interaction step actually did,
• and artifact.html itself, to read.

The judge gets that folder read only and scores exactly what it can see in there. No “well, a model like that usually does fine”.

The earlier version took two screenshots and that was not enough. Three things kept slipping through:

The frozen first frame. A simulation that renders one frame and dies looks identical to a working one on a single screenshot. Only the 1s / 6s / 12s spread shows whether anything is actually moving. The reverse matters too: on a page that is meant to be static, three identical shots are not a defect – and the judge is told that explicitly in the prompt, so it doesn’t punish a page for not blinking.

Everything below the fold. A model builds a nice hero and forgets the rest, or the other way round: a bland first screen with all the real work further down. Judging looks from a single viewport is a lottery, so the look criterion is scored from shot-full.png.

Buttons that do nothing. That was the worst case. The page loads, the console is clean, it looks finished, you click – and nothing happens. You cannot see that in a screenshot.

Clicking blind is harder than it sounds

The interaction step took me the longest, because it forces you to answer one question: what exactly do you click, when you have no idea what the model generated?

The default scenario clicks the first three controls that pass a clickability check. With two caveats:

• Real controls before anchors. <button>, [role=button], submit, summary come first, href="#" links only after them. An anchor just scrolls, so if the nav links ate all three slots I would have proven nothing.
• DOM order. Not random, not “the one that looks most important” – simply the order in the document. That way the same page always yields the same choice and the run is reproducible.

“Visible and enabled” is not enough. A button inside a closed modal satisfies both while sitting under an overlay, and clicking it only times out. Playwright has click({ trial: true }) for exactly this: it runs the full actionability check (visible, stable, receives pointer events) without firing the event. The same bar a real click has to clear.

Two tasks have their own scenarios, because clicking would prove nothing there: snake (a keyboard game – you have to send arrow keys) and todo-app (an empty list – you have to type something and add it first). And interactions.json also records the steps that found nothing – “the page has no working interactive element #2” is evidence too, and strong evidence at that.

Two judges, blind, with mandatory rationales

The judges are Claude Code and Codex, each run as a CLI in read-only mode. A few rules I consider more important than the choice of model:

• They score the same criteria I do (compliance, works out of the box, look, code quality) plus two I don’t track by hand: code structure and logic correctness.
• Blind. A judge never sees my scores or the other judge’s scores.
• Values are snapped to the criterion’s scale. If the scale is 0 / 0.5 / 1, there is no “0.7 because that’s how I feel”.
• A rationale is mandatory for every 0 and 0.5. It has to name a concrete defect, not “could be better”. At the top value the rationale is optional – and that’s deliberate, because what I care about is why something is bad.
• The median across judges is computed in the viewer and never lands in the file. The data holds raw scores; the aggregate is a presentation concern.
• Divergence gets a badge. If the human and the judges differ by at least 0.5 on a shared criterion, the table lights up. It’s a list of things for me to look at, not an alarm.

There is a separate stability judge: it takes all attempts of one model on one task, computes score variance and code similarity (token Jaccard), and adds a written verdict. It needs at least two attempts.

The part I didn’t expect

Once the judges had chewed through the whole set, I sat down and compared it with my manual scores. The material: 381 attempts, 4 tasks, 16 models, two judges – 762 machine verdicts against 381 human ones. I only compare criteria we both score (agent logic is out; the judge never sees the agent’s process, only the finished artifact).

And here is the boring but important distinction that is easy to miss: it looks one way per model, and quite another way per individual attempt.

Per model, we agree closely. Averaging all attempts of a model and comparing against the judges’ average:

The average difference per model is 9.8 percentage points, the largest 18.4. Rank correlation (Spearman) comes out at 0.90 – the top and the bottom of the table agree, the middle reshuffles: Qwen 3.5 122B drops from third to seventh in the judges’ ranking, Gemma4 31B climbs from sixth to third.

Per single attempt it gets a lot messier. Same measure, computed per file:

• average difference: 15.8 pp, median 11.5
• 38% of attempts land within 10 pp – but 19% diverge by more than 30 pp
• record: 86 pp on a single attempt

So averaging over six attempts eats noise that is enormous on any individual file. The model ranking is shared; the score of one specific artifact can be completely different.

Two things that fell out along the way and changed my picture:

The judge is systematically harsher than me. In 232 of 381 attempts (61%) it scores lower, on average by 9.8 pp. That’s not noise, that’s an offset. I score “runs and looks fine”; the judge gets an evidence folder and lists every console error.

The judges differ from each other about as much as they differ from me. Claude and Codex diverge by 10.2 pp on average (record 42.3). So this isn’t “human versus machines” – it’s three graders with a similar spread.

Now the interesting part. Broken down by task:

That’s the answer to my own question. I had three hypotheses for why we agreed so much:

1. On small, well-scoped tasks “is this good” simply isn’t ambiguous enough to disagree on.
2. After a dozen-plus years you spot certain things on autopilot – and then I score the way these models were trained to score, so these aren’t three independent opinions, just three versions of the same school.
3. I’m testing the easy end.

The data says clearly: mostly the first and the third. Todo app – 4 points of divergence, near-perfect agreement. The page with a long spec – six times the divergence, with the judge consistently 22 points below me. The more there is to check, the further apart we drift. The agreement that surprised me was, to a large extent, agreement on easy tasks.

The divergence also concentrates in specific criteria: most often look (125 cases) and compliance (89), less often “works out of the box” (44). So we argue about inherently judgement-heavy things and about ticking off a long requirements list, not about whether the page starts at all.

A vision model can judge how something looks. And that is exactly the moment to be careful

The thing that keeps nagging me: vision models really can do this. Give them a full-page screenshot and they’ll catch a heading overlapping the text, terrible contrast, a layout falling apart at the third section. Back in the GPT-3.5 days this would have been unthinkable. Today the judge finds visual defects I would have missed at 23:40 on the seventy-fifth file.

And here’s the catch.

You can perfectly well build a loop where one model writes the page, a second one scores it, a third one fixes it according to the notes – and no human ever looks inside. Technically it works. It’s tempting, because it’s fast and it scales.

Except “looks good” is not a property of an HTML file. It’s a statement about how something works on a human. Readable to whom? Comfortable for whom? A model judge is a good approximation of that statement as long as it stays anchored in human judgement – in the data it learned from, and in someone regularly checking whether they still agree with it.

Unplug that anchor and after a few turns of the loop you’re left with a system optimizing something that used to be human taste and is now its own drifted version. Things start being made for the grading model instead of for the user. That’s the moment we’re in trouble – not because AI rebels, but because nobody notices when the measure stopped measuring what it was supposed to.

So in my setup the judge is an advisor, not the final judge, its name notwithstanding. It takes the grind of 381 attempts off me and tells me where to look – and, as you can see above, it sometimes catches me scoring things too quickly. I still sign off the ranking myself.

All the results are here: benchmark.refio.dev. If you run your own eval harness – tell me whether you see the same pattern: agreement on small tasks, divergence growing with the length of the spec. I’m curious where that line sits for other people.

That’s it for today :P