wartzarlab

Field note from a few weeks of running agent-mode on a large-ish repo: the thing that quietly wrecks a long session isn't a bad edit, it's context bloat you can't see. The agent keeps pulling more files in, the token count creeps up, and by the time it feels sluggish or starts dropping earlier instructions you've already burned the budget.

What actually helped me was making the cost visible before the run instead of after. Two cheap habits:

1. Budget the prompt, not just the diff. Before kicking off a background agent on a task, I do a rough token count of what it's likely to load (the files I @-mention + their obvious imports). If the "starting context" is already a big fraction of the window, the run is going to degrade — so I split the task instead of hoping.

2. Attribute tokens to files. When a session gets expensive, the culprit is almost always one or two fat files (a generated client, a giant JSON fixture, a barrel index that drags in everything). Counting tokens per included file — not per message — points straight at them. Usually the fix is "don't feed the agent the generated file, feed it the schema."

The mental model that stuck: treat the context window like a CI cost gate. You wouldn't merge a PR that 10x'd your build time without noticing; same idea for the tokens an agent pulls per task. A number you can see at the start changes the plan; a number you discover at the end just explains the failure.

Curious how others here keep long agent runs from silently outgrowing the window — do you prune context manually, split tasks, or just restart the session when it gets heavy?

A habit that saved me a lot of failed local runs: I started measuring my context budget before a run instead of discovering the ceiling when generation silently truncates or the KV cache OOMs my GPU.

The thing that finally clicked for me is that on local models the context window isn't just a quality knob — it's a hard memory bill you pay up front. KV-cache size scales with (context length × layers × heads × 2 × dtype bytes), so doubling the prompt you feed in can quietly double the VRAM you need for cache before a single token is generated. On a 24GB card that's the difference between a run that fits and a run that pages/OOMs mid-generation.

What I do now, in order, before a long-context run:

1. Tokenize the actual prompt, not a guess. The system prompt + retrieved chunks + few-shot examples + chat history almost always adds up to more than I expect. I count them with the model's own tokenizer, because token/word ratios drift a lot between a code-heavy prompt and prose.
2. Budget the reply too. n_predict/max_tokens reserves cache. If I want a 1k-token answer I need room for prompt + 1k, not just the prompt.
3. Compare against the real ceiling I loaded with, not the model's advertised max. If I loaded llama.cpp with -c 8192, that 8192 is my wall regardless of what the model card says it can do — and rope-scaling to a bigger context has its own quality cost.
4. Trim at the retrieval/history layer, not by truncating the front of the prompt. Silent left-truncation is how you lose the system prompt and get a model that "forgot its instructions."

The mindset shift: treat the context window like a resource you allocate deliberately, the same way you'd think about VRAM for weights. Once I could see the budget as a number before hitting enter, a whole class of "why did it cut off / why did it OOM / why did it ignore my instructions" problems just went away.

Curious what everyone else uses to keep an eye on this — do you eyeball token counts, script it, or just crank -c and hope the card holds?