One small trick and you stop losing context in local LLM models

Recently I was messing around with the Refio plugin and testing it on local LLM models. In theory everything worked: the model was supposed to get the whole project context – open files from the IDE, results from the RAG, the rules of engagement from the system prompt. And… nothing. It didn’t work.

Instead of accurate answers – vague generalities, as if the model couldn’t see a thing. And it’s not like it was running out of tokens: the limit was set to 32k, and the context fit comfortably below that line.

The problem: a model as blind as a developer after a few too many

The scenario looked like this:

1. The user drops @open_files from IntelliJ into the context – the plugin checks which files are open and glues them onto the context. That comes out to 8k+ tokens.
2. They ask a question about those files.
3. The model answers something about opening files. :P

The model ignored the context. Or maybe it “saw” something – from its general knowledge – but it definitely wasn’t answering on topic.

Let’s split it up – the idea that does NOT work (on local models)

At first I thought: maybe it’s a prompt structure thing?

I tried the classic separation:

1. system: Policies ("You are an AI assistant...", tool descriptions, etc.)
2. system: Context (code, RAG, open files)
3. user:   question

Sounds reasonable. And the result?

Nothing.

The model kept blowing off the context. Or reading it like a washing-machine manual in Hungarian – that is, not at all.

Fun fact: the exact same approach worked great on cloud models. OpenAI, Anthropic, Claude via OpenRouter – they handled the separation of policies and context without a hitch, and ignoring the context basically never happened. So the problem wasn’t the prompt itself, but in how local models cope with long context. And that’s where the more interesting part begins.

What the research says

I started digging and it turned out this isn’t my personal pet peeve, it’s a measured phenomenon. Three things worth knowing:

Curve from the “Lost in the Middle” study – accuracy depends on where in the context the information sits. The model holds onto the start and the end, the middle gets blurry. Slide from the talk “A Million Tokens Later”.

1. Lost in the Middle (Liu et al. 2023) – models hold onto the start and the end of the context best, while the middle gets blurry on them. It’s a U-shaped curve: a key detail tucked into the middle of a long prompt can be practically invisible. The model isn’t really “losing” it – it just doesn’t effectively use it. A bit like you with a 30-page terms-of-service: first line, last line, “I accept”.

2. RULER (Hsieh et al. 2024, NVIDIA) – a benchmark that doesn’t pat models on the head, it checks how much context the model actually handles. The result: out of 17 models only half hold their quality at 32k – and that’s a short window for them. The advertised 128k or 200k is a promise of capacity, not a guarantee of quality across the whole length. Like “up to 1 Gbps” from your internet provider.

3. Context Rot (Chroma 2025) – quality drops with input length even on a trivial task, not just when hunting for a needle in a haystack. What matters is not only how much context you throw in, but how you arrange it.

The conclusion for me was simple: since the model remembers best what’s close to the end, there’s no point in leaving only the user’s question down there.

A simple trick: change the order

Instead of:

1. system: Policies
2. system: Context
3. user:   question

I use:

1. system: Context   - even 8k-20k tokens
2. system: Policies  - right before the question!
3. user:   question

Yes, this deliberately breaks the usual advice of “policies at the very top”. On the cloud that advice works, because those models have a bigger effective window. Locally – it gets in the way. The context lands at the start (the model will scan it like chapter one), and the rules and answer format sit right before the question – exactly where the model’s attention is freshest.

It’s like studying for an exam: you don’t remember the whole book, but the last two pages you read right before walking in – those stick in your head.

Why this works

Local models have a limited window of real attention. They can theoretically take in 32k or 128k tokens, but that doesn’t mean all of them are equally important – see the U curve from Lost in the Middle. By reordering, you put the most important things (rules + question) into the zone the model holds onto best.

TL;DR – for the busy

1. Marketing lies – the real handling of long context is worse than the advertised 32k or 128k (RULER).
2. Order matters – the model sees the whole context, but doesn’t use it evenly; the middle gets blurry (Lost in the Middle).
3. Recency bias – three lines of change, and a huge effect.
4. Cloud ≠ local – what works great on OpenAI or Claude can be useless offline.

PS. The mandatory benchmark

If you work with local models – don’t trust the documentation. Fire up RULER, run the benchmark, see where your model falls apart. Better to find out early than to spend two days getting frustrated over a prompt that’s supposedly good.

This is a thread from my talk “A Million Tokens Later” (deBBug, JUG Bielsko-Biała). There I go into the whole thing: why a big window isn’t memory, how the KV cache eats your graphics card, and how Refio builds context in layers. Slides and measurements – czub.info and benchmark.refio.dev.