Code that kills the browser – observations from a small LLM benchmark

There’s a moment when the model returns code, the file opens, and for two seconds you think it works. Sometimes it actually does. Other times the browser tab goes blank, or you get a white screen with an error in the console. Same model. Same task. Different run.

This is a story about the most boring and most important quality of LLM-generated code: consistency.

The task: one HTML file, a neural network

I added a task to the benchmark I use for testing code-generation models (https://benchmark.refio.dev/tasks/neuron_growth_simulation): write an interactive neural network growth simulation in a single HTML file. Canvas, simple 2D physics, particles, a few sliders. No libraries, no backend – it just has to work.

I tested 16 models, six runs each – cloud models (Claude, GPT, GLM) and local ones running on my own machine (Gemma, gpt-oss, Qwen in various sizes). I grade every artifact with the same criterion I call works out of the box: does it run and do what it was supposed to, without manual fixes. Zero = doesn’t start, half = works but with bugs, one = works.

Results: not “who can do it”, but “how often it pulls it off”

Top and bottom of the table (full table with all 16 models at https://benchmark.refio.dev/tasks/neuron_growth_simulation):

(Qwen 3.5 9B ties with Gemma 4 26B at 0.25; the full sixteen are on the benchmark page.)

Two things stand out.

First: only two models went six for six – Sonnet 4.6 and GPT 5.1 codex mini. The entire top of the table is cloud models and large models.

Second, and this is the more interesting part: no model scored zero out of six. Even the weakest Gemma 4 31B, which failed to start five times, produced something working once. So the difference between models isn’t “this one can, that one can’t”. Every model can write this. The difference is how many times out of six it actually does.

And that’s exactly the quality you can’t see when you run a model once, hit a good run, and say “works great”. A small local model that produces working code one out of four tries looks identical to the one that does it every time – in a demo. The difference only shows up at scale – meaning in real work.

What “doesn’t start” actually means

The most common reasons a run landed in the “doesn’t start” column were mundane:

• White screen – code loads, but nothing appears on the canvas (typo in a selector, a loop that never kicks off).
• Console error on start – classic undefined is not a function, because the model referenced something it never defined.

Nothing exotic. Just code that looks like code, but isn’t code that runs.

The other kind of failure: when the code doesn’t crash, it just hangs

The benchmark catches runs that don’t load or have bugs. But there’s another, nastier failure mode that a dry “works / doesn’t work” score doesn’t capture: code that loads, starts – and after a few seconds freezes the tab.

A growing network simulation is a perfect trap for this. You do the animation via requestAnimationFrame, which calls itself in a loop. If the “growing” network grows without a cap, and drawing connections is quadratic (each node with every other node), then after a few hundred nodes a single frame takes seconds to compute, memory bloats, and the browser hangs – on its own. The code technically “works”.

This isn’t theory. I opened one of the gpt-oss 20B runs (local model). The heart of the simulation is an update() function that requestAnimationFrame calls about 60 times per second. And inside it, something like this:

// gpt-oss 20B, run 02 - fragment of update(), called every frame
finishedBranches.forEach(b => {
  if (b.depth >= params.maxDepth) return;
  if (Math.random() < params.splitChance) {            // a finished branch rolls "should I split?"...
    const child = new Branch(b.end, dir, b.depth + 1); // ...and does this EVERY FRAME, forever
    neurons.forEach(n => n.branches.push(child));      // and appends the same branch to EVERY neuron
  }
});

// below, in the same frame - synapse detection "everyone with everyone" over the growing list:
for (let i = 0; i < finishedBranches.length; i++)
  for (let j = i + 1; j < finishedBranches.length; j++)
    if (nearby(finishedBranches[i], finishedBranches[j])
        && !synapses.some(s => ...))   // and inside, yet another scan through all synapses
      synapses.push(new Synapse(...));

Look at what’s happening here. A finished branch never leaves finishedBranches. So every frame, every branch that ever existed rolls the dice on whether to multiply – and the winners spawn new branches. The list grows, and new children get appended to all neurons at once (that part’s just a plain bug). On top of that, the synapse loop: two nested for loops over the same bloating list, with synapses.some(...) inside. At a hundred branches that’s ten thousand comparisons per frame. At a thousand – a million. Every frame. No node cap, no cancelAnimationFrame (I checked – it’s not in any of the 96 runs).

The result is exactly what I described at the start: it loads, grows nicely for a moment, then one frame takes fractions of a second, then seconds, RAM bloats. The tab doesn’t throw an error – it just stops responding. Sometimes the whole browser goes with it (process isolation in theory, but…).

The best part: one of the models (Qwen 27B) actually left a comment right there: “For now we keep them to maintain the network structure” – it noticed that nothing gets cleaned up, decided that was a feature, and moved on. It signed its own memory leak with a comment :)

For contrast, Haiku – which passed this task almost every time – did one simple thing: the split fired once, at the moment a branch finished growing, and only while depth < maxDepth. The tree closes, the network is finite, the loop has something stable to draw and doesn’t eat the machine. Same animation, same slider library – the difference is one “when”.

Takeaway

“Compiles” is not the quality bar – it’s the floor. The bar is consistently working. And that’s the whole point of running your own benchmark instead of reading other people’s tables: you only see that the weakest model can have a great day, and you recognize the best ones by the fact that they don’t have bad ones.

I test models through a plugin for IntelliJ and CLI at https://refio.dev/ – my sandbox for testing local and cloud models on Agent tasks, which doesn’t exactly work the standard way. If you’re digging into similar stuff, reach out – happy to talk.