Last time I wrote about the judge that scores my benchmark, I left one thing out. That post was about adding a model to the scoring. It was not about the fact that the model is the last thing I reach for in this machine.
Evaluating an AI coding model is not one number. It is several different questions: did it meet the spec, does the thing actually run, does it look good, is the code clean, and did the agent behave sensibly. Each of those questions has a different natural judge. So in Refio I split scoring into three layers: a deterministic judge (measures what a machine can measure), strong LLM judges (read the code and score what a machine cannot weigh), and a human (visual taste and the final call). The whole thing runs on the same agent engine the production plugin uses, just headless. The result: model comparisons you can actually trust, because they are multi-dimensional, repeatable, and reflect how the tool really behaves.
“Works on my machine” does not scale to 480 runs
Anyone who has plugged a new model into their agent knows the moment. You fire off a prompt, the model spits out an app, you open it in a browser and say “huh, not bad” or “nope, garbage”. You try again and get something different. You plug in the next model, and an hour later all you have is a vague “the first one was probably better”. That is not evaluation. That is guessing in good faith.
Leaderboards do not save you, because they measure something else: how a model handles closed, self-contained puzzles. They do not tell you whether your agent, with your tools and your context budget, will get something out of it that opens and does not flood the console with errors. I have that problem in a sharp form while building the plugin, because Refio only makes sense if I can honestly say: this local model, free, on your GPU, is good enough for this kind of task, and that one is not.
Why one number lies
Take a real task from my catalogue: a simple 3D editor in the browser, single HTML file – scene view, adding and moving objects, a properties panel, zero console errors. How do you score that with one digit? You cannot, because several independent questions live inside it:
- spec compliance – can you actually add an object, is there a properties panel, does the file name match,
- works out of the box – does it open without manual fixing, does it throw errors,
- look – this is taste, not measurement,
- code quality and structure – clean, or a tangle with dead functions,
- logic correctness – does it hold up when you read the code. A screen can lie beautifully,
- agent logic – did the agent check files, generate, verify, summarise. Or did it wander.
A model can crush one dimension and faceplant in another – a great look on top of broken logic is a daily occurrence here. On top of that comes non-determinism: the same prompt twice gives you two different files.
The principle this stands on
Do not judge everything with one tool. Do not call a model where a plain function will do, and do not pretend a regex can weigh taste.
Look at the extremes. “Does this page render?” has a binary, machine-checkable answer; asking a model “do you think it renders?” injects noise where you could have had certainty. Just render it. In the other direction, “does this look good?” is an aesthetic judgement by definition – a regex will not weigh it, and even a strong model does it worse than a human with taste.
Hence three layers.
Layer 1: the judge with no opinions
The first layer has nothing to do with AI, and an important caveat up front: it is a hint, not a result. Those scores sit next to an entry while I am looking at it, and that is where their role ends – they never make it into the final results. It measures three things you can compute objectively from an e2e run.
Compliance through needles. For every task I define a set of needles – content markers that must appear in the delivered file. All of them hit is 1, some is 0.5, none is 0. A needle is either plain text or a regex, so the rule is hard and repeatable. This is where people usually ask “but there are no unit tests here, how do you measure anything?” – exactly like this. When the artifact is an HTML file rather than a function with a signature, you cannot write classic tests, but you can say what must be in it.
Works out of the box through an actual render. The file goes into a headless browser. Renders clean – 1, renders with console errors – 0.5, does not render at all – 0. Zero interpretation. This layer grew directly out of what I wrote about in the code that kills the browser: once you have seen an infinite draw loop lock up a tab, you stop trusting a screenshot as proof that something “works”.
Agent logic through status and tool order. The run has to finish with a SUCCESS status, and if a task requires a specific tool order, I check whether it is a subsequence of the actual calls – they do not have to be adjacent, but the order has to hold. SUCCESS with the right order is 1, with the wrong one 0.5, no SUCCESS is 0.
All of this is free, instant, and one hundred percent repeatable. Before I look at anything, I already know whether the page came up and whether the agent took the right path. And importantly: it deliberately does not touch look or code quality, because those cannot be measured mechanically.
Layer 2: models, but only now
The second layer takes what a machine cannot weigh but a careful read of the code can: code structure and logic correctness. This is where judgement needs intelligence, so this is where I finally call a model (through the Claude Code and Codex agents). I described the construction in the previous post – an evidence folder instead of “rate this 1 to 10”, screenshots at three points in time, clicking through controls, blind scoring. Two things follow directly from the layering.
I aggregate with the median, not the mean. The median shrugs off one judge who went off the rails; the mean politely folds that excursion into the result. The known sins of model judges (verbosity bias, position sensitivity, inconsistency across runs) do not disappear, but they stop deciding once you have two of them and take the middle. Which is why the judge here is an advisor, not a final authority – it tells me where to look, and the ranking is still signed by me.
LLM judges do not score agent logic. Not because they cannot – because how the agent worked live cannot be reconstructed from a dead file. That is a question about process, not product. A good evaluation system knows what its judges should not touch.
Layer 3: the human, trimmed to the minimum
The last mile is mine, and I do exactly one thing in it that cannot be automated: I add look (the only criterion on a 0-2 scale, because that is where “better than I expected” belongs) and make the final call.
An automated step assembles a ready entry in a review queue – artifact, screenshots, duration, tokens, cost, deterministic hints, judge verdicts and a tentative pass/fail. I walk in, look, add my part, and either promote the entry into the results or reject it. Nothing that could be computed gets retyped by hand. On top of that there is a divergence marker: it shows the largest gap between my score and the judges’ aggregate. It works both ways – either the judge got it wrong, or I missed something it caught while reading the code.
Repeatability, and where the tasks come from
Since models are non-deterministic, one good result can be luck, so I measure stability separately. For me that is six runs per model per task – fewer than three and you cannot tell luck from form, more than six and it stops paying off across a dozen-plus models. For any group (task, model, environment) with at least two runs I compute the variance of scores and the similarity of the generated code. Incidentally, that second measure turned into its own post – it turns out every LLM writes the same Todo App, quite literally.
The tasks themselves have a single source of truth: one description plus one prompt, from which a generator emits both the automated e2e scenario and the review task in the admin panel. I learned this the hard way – keep a task description in two places and a month later you have two different tasks under one name. Difficulty is spread on purpose: from Snake and Todo, through the 3D editor, up to single-file monsters like a modular Web Audio synth, a logic circuit simulator or a procedural voxel world. That is where weaker models simply break, which is the whole point.
The same engine, just without the windows
The most important part last. The benchmark does not call the model through a clean API under laboratory conditions – it drives it through exactly the same agent engine, the same tools, the same context pipeline and the same guardrails the production IntelliJ plugin uses. Just headless.
So I measure the model the way you will actually experience it in Refio: with its tool-use discipline and its behaviour inside the agent loop. That makes the benchmark useful twice over – it scores models, but it is also my own proving ground. When a guardrail kills a sensible run or a model falls into a loop, I see it here before anyone else does.
What this buys me
- Honesty. A model cannot hide broken logic behind a pretty surface, because those are two criteria judged by two different judges.
- Repeatability. The hints I start every entry with are deterministic, and stability is measured explicitly.
- Realism. I measure the model in the same engine you use it in.
- Cheap decisions. I drop in a new local model and know by the end of the afternoon whether it is worth shipping.
The whole philosophy fits in one sentence: deterministic where you can, model where you must, and a human for taste. Boringly simple, and it turns “the first one was probably better” into numbers I can trust.
Results live here: benchmark.refio.dev. If you are building your own evaluation harness, I am curious where you drew the line between “I will compute this myself” and “let a model score it”. My hunch is that most people jump straight to layer two and pay for it in noise.
0 komentarzy