Model Compaction

Wed, 11 Nov 2026 00:00:00 +0000

this article is written fully by me, a human - no ai’s generated any text here.

if you’re working with ai or building an ai harness then you’re probably aware of the context limits that models have. You can’t chat with a model infinitely and have it remember everything perfectly, but all popular chat apps seemingly do let you chat infinitely, so how do we get around the technological limits of model context windows?

what is compaction?

let’s say you’re building a harness around one specific model: claude fable 5.2. Anthropic says that fable52 has a context window of 1 million tokens, which means you can provide up to 1m tokens in every request before anthropic starts rejecting your request with context length exceeded errors. but when you’re running a lot of turns with an agent, executing a lot of tool calls with large outputs and consuming large files, you can quickly reach that 1m limit. so how do harnesses deal with this issue?

compaction is the process of reducing the size of your session so that you fit within the context window again and can continue working. when using 3rd party harnesses like OpenCode, Pi, Claude Code, etc, will have their own implementation of compaction built-in so you don’t have to worry about it. each harnesses implementation of compaction differ slightly and that will impact your results in large sessions.

when does compaction happen?

it’s up to the harness to decide when to compact, so if you’re building your own harness then it’s up to you, but a lot of harnesses choose to compact when the total tokens in the sessions are within some threshold of the limit. so for example, you can trigger a compaction when your tokens reach 75% of your context window for your model, which means when you reach 750k tokens out of 1m. usually a buffer is provided because output is counted against the window limit. this check for tokens can be done at the beginning of a new turn, mid-turn if you happen to surpass the threshold because of large tool call outputs during the run, end of turn, manually by the user, or even after being idle for some time. claude code, for example, can compact a large session after about an hour idle, right before the 1-hour prompt cache would expire, so your next message doesn’t have to reprocess the whole history uncached.

compaction strategies

this is the most interesting part to me. how do harnesses compact? what exactly is happening during this process? well, it varies by harness, but the vast majority of harnesses will compact the session by doing another llm call to summarize the session. during summarization, the model takes a customized prompt that explains that it needs to compact and it takes the history of the session and it tries to output a much shorter version of what happened and tries to keep the important bits.

there are a lot of things you can do here before calling the summarization model call. before we get into them though, let’s recap what’s part of a typical session. in a normal session you typically have system prompts, assistant messages, user messages and tool calls. tool calls can contain things like skills that are loaded, or they can just contain unnecessary things that are no longer needed. before calling the llm to summarize you can choose to delete all tool calls for example, and that will result in a cheaper and more accurate summarization step. it depends on what your harness is built for though, if you don’t want your model repeating expensive tool calls again then you can persist those post compaction so future requests still see them.

summarization in the form of an llm call at a trigger point is a popular strategy for compaction, but it’s not the only strategy. some systems decide to keep a running notebook of important items so when it’s time to compact, instead of doing a summary call to the llm, it just persists the notes from the notebook and continues on (credit to Robin Salimans, Zapier). this could work well for your use case but of course it depends on the results of your own evals.

Below is an interactive example of what compaction can look like on a real session. try moving the sliders around to see when and how the compaction is affected.

Turns in session: 4 / Compaction threshold: 75% of window Keep recent tokens: 50k

Before no compaction

After with compaction

recall and history

compaction is inherently lossy, meaning you’ll lose some information during the summarization step. losing some non-critical information may be fine for your use case, but usually we want to provide future model calls the ability to make tool call requests to recall data from before the compaction took place. this can be optimized by giving future calls some sort of table of contents to the pre-compacted conversation with references to jump to specific topics. this recall ability provides the model a way to lazy-load more context on demand instead of needing to keep it all in the context live. if your harness is running in an environment where you can provide is a file system or virtual file system, then another good strategy for recall is dumping important tool call results and past conversation directly to files on disk and providing your harness the ability to search through and parse these files for context.

conclusion

context compaction is a really fascinating topic. for further reading i recommend checking out these papers that dive deeper into it:

Ai on Jason's Blog