Idle Compression in Agent Harnesses

TL;DR

• The Problem: Coding agents easily accumulate 200K–300K tokens of tool outputs. Prompt caching keeps active turns cheap and fast, but provider caches expire after a short Time-To-Live (typically 5 minutes). Stepping away for coffee causes a cold-cache prefill on your next message—spiking latency and costing up to $10\times$ more.
• The Solution: TTL-Aware Idle Compression starts a background countdown after each turn (e.g., at ~4.5 minutes). Right before the cache expires, the harness appends a summarization prompt to the existing conversation—reusing the still-warm KV cache (~95% cache hit)—and shrinks the bloated history down to a lean ~19K-token context.
• Zero Disruption: It runs speculatively during human think-time. If you start typing while compression is mid-flight, the harness immediately aborts the background job and preserves your active flow with zero blocking latency.
• The Trade-off: Compaction is lossy and speculative. Summarizing drops verbatim details (potentially forcing file re-reads), and compacting an abandoned terminal tab burns output tokens on a session that never resumes—making its ROI sensitive to cache pricing cliffs and session resume rates.

The blog is co-authored with Gemini Carbon.

If you have spent an afternoon pair-programming with a terminal-based coding agent—such as Claude Code, Qwen Code, or OpenClacky—you have likely witnessed how fast an agent’s context window balloons. Within twenty minutes of grepping files, reading stack traces, running test suites, and iterating on patches, a session effortlessly crosses 200K to 300K tokens.

While you are actively conversing with the agent, prompt caching makes that massive context feel fast and affordable. Every turn reuses the cached Key-Value (KV) state from previous turns, charging a fraction of standard input token rates and keeping Time-To-First-Token (TTFT) low.

Then you step away for ten minutes to grab a coffee or review a pull request. When you return and type “looks good, now update the unit tests,” two painful things happen at once: latency spikes, and that single turn quietly costs up to $10\times$ more than the turn before it.

Idle compression is the harness-level architecture designed specifically to eliminate this hidden tax.

1. The Core Problem: The Prompt-Cache Cliff

Modern LLM APIs rely on prefix-based prompt caching with a strict Time-To-Live (TTL)—most famously, a standard 300-second (5-minute) ephemeral cache window. As long as each new request hits the API before that timer expires, the cached prefix stays warm and its TTL resets.

flowchart LR
    subgraph s1["❌ Without Idle Compression — Hit the Cache Cliff"]
        a1["1. Active Coding<br/>250K context (Warm)"] -->|"Break > 5 min"| a2["2. TTL Expires (300s)<br/>KV cache evicted"] -->|"User resumes"| a3["3. Cold Prefill 250K<br/>~312.5K eq. · Slow!"]
    end
    subgraph s2["✅ With TTL-Aware Idle Compression — Beat the Clock"]
        b1["1. Active Coding<br/>250K context (Warm)"] -->|"Idle hits 266s"| b2["2. Warm Compact<br/>95% hit · 250K → 19K"] -->|"User resumes"| b3["3. Cold Prefill 19K<br/>~24K eq. (−92%) · Fast!"]
    end
    style a1 fill:#eef6ff,stroke:#3b82f6,color:#1e3a8a
    style a2 fill:#fff7ed,stroke:#f97316,color:#7c2d12
    style a3 fill:#fef2f2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px
    style b1 fill:#eef6ff,stroke:#3b82f6,color:#1e3a8a
    style b2 fill:#ecfdf5,stroke:#10b981,color:#065f46,stroke-width:2px
    style b3 fill:#ecfdf5,stroke:#16a34a,color:#14532d,stroke-width:2px

Let’s look at the token economics of a 250K-token coding session when cached reads cost 10% of base input pricing ($0.1\times$) and cache writes cost $1.25\times$:

• Warm Turn (Active Session): Re-reading 250K cached tokens costs the equivalent of just 25K base input tokens, plus your tiny new message.
• Cold Turn (After a 10-Minute Break): The 300-second TTL has expired. When you send your next prompt, the provider must re-prefill all 250K tokens from scratch—often charging a cache-write premium ($1.25\times$ base input cost, or 312.5K equivalent tokens). That single resume turn costs more than twelve warm turns combined, accompanied by several seconds of cold-prefill latency.

Most of those 250K tokens aren’t even durable architectural knowledge. They are ephemeral scaffolding: raw git diff dumps, bulky JSON logs, failed compiler outputs, and files the agent inspected once and discarded. Paying full cold-start prices to re-ingest stale terminal noise is pure waste.

2. Why Traditional Compaction Falls Short

Historically, agent harnesses managed context bloat using mechanisms that fail to address the cache cliff cleanly:

1. Reactive Auto-Compaction (Token-Ceiling Trigger): The harness waits until the conversation hits a hard threshold—say, 80% of the model’s maximum context window—and then pauses to summarize the history.
   • Flaw: It runs on the critical path. Just when you are deep in flow debugging a tricky issue, the terminal freezes for 15–30 seconds while the agent compacts its memory. Worse, if your session sits at 65% capacity and you take a break, reactive compaction never triggers, leaving you exposed to the cold-cache penalty.
2. Naive Long-Idle Compaction: Some early harnesses scheduled background cleanup after 30 or 60 minutes of user inactivity.
   • Flaw: By minute 30, the 5-minute prompt cache is already dead! Running a summarization LLM call at that point forces a 0% cache-hit cold prefill across the entire bloated conversation just to generate the summary.

Wait—Why Not Just Strip Tool Outputs Lazily After Cache Expiry?

A sharp systems question arises here: if the KV cache is already dead after 5 minutes anyway, prefix matching no longer matters—so why not wait until the user returns at minute 10 and deterministically delete old bash and read_file outputs in memory for zero LLM tokens?

Some harnesses do implement this lazy deterministic pruning (sometimes called micro-compaction). While a cache is warm ($t < 5\text{ min}$), mutating historical messages in the middle of the transcript is forbidden because it invalidates the KV prefix for every subsequent turn; once the cache expires, stripping raw stdout payloads on resume is completely free.

However, blind truncation deletes conclusions along with noise. If the agent ran twelve commands to isolate a subtle race condition, wiping those tool outputs erases what the agent learned, forcing it to re-investigate from scratch. TTL-aware idle compression solves the harder problem: distilling 200K+ tokens of raw execution traces into semantic engineering state—and doing so requires running an LLM pass before the warm KV cache evaporates.

3. How TTL-Aware Idle Compression Works

TTL-aware idle compression treats context compaction as a cache-deadline scheduling problem. Instead of waiting for the context window to overflow or waiting until the user has been gone for an hour, the harness races the cache TTL clock.

Step 1: Arming the Sub-TTL Watchdog

The moment an agent finishes its turn and yields control back to the user prompt, the harness starts an asynchronous idle timer set slightly below the provider’s cache TTL:

$$T_{\text{trigger}} = T_{\text{TTL}} - \Delta_{\text{safety}}$$

For a 300-second cache TTL, harnesses typically choose a safety margin $\Delta_{\text{safety}}$ of 30–40 seconds (for example, triggering at 266 seconds). This gives the request enough headroom to reach the inference cluster before the prefix cache gets evicted, while ensuring compaction doesn’t fire prematurely while the developer is simply reading code and thinking.

Step 2: Prefix-Preserving “Insert-Then-Compress”

A classic mistake when implementing summarization is sending the conversation transcript to a separate helper prompt with a brand-new system instruction ("You are a helpful conversation summarizer..."). Because KV caches match strictly from token 0, changing the system prompt invalidates the entire prefix cache!

Instead, production agent harnesses use insert-then-compress, keeping the exact prefix intact and appending the compaction instruction as the final user turn:

flowchart LR
    subgraph bad["❌ Anti-pattern: separate summarizer prompt (0% cache hit)"]
        b1["NEW summarizer<br/>system prompt"] --> b2["Serialized<br/>250K transcript"]
    end
    subgraph good["✅ Production: insert-then-compress (~95% cache hit)"]
        g1["Agent system<br/>prompt + tools"] --> g2["250K live history<br/>(warm KV prefix)"] --> g3["Appended compact<br/>prompt (~500 tok)"]
    end
    style b1 fill:#fef2f2,stroke:#dc2626,color:#7f1d1d
    style b2 fill:#fef2f2,stroke:#dc2626,color:#7f1d1d
    style g2 fill:#eef6ff,stroke:#3b82f6,color:#1e3a8a
    style g3 fill:#ecfdf5,stroke:#16a34a,color:#14532d,stroke-width:2px

Because the entire prefix matches the warm KV cache from 4.5 minutes ago byte-for-byte, the summarization call itself achieves a ~95% cache hit rate, making background compression remarkably cheap and fast.

Step 3: Surgical Context Replacement

Once the model returns the structured summary, the harness rewrites the in-memory session log:

flowchart LR
    subgraph before["BEFORE — 250,000 tokens"]
        p1["System + tools<br/>8K"] --> p2["Turns 1–42: raw reads, diffs, logs<br/>236K"] --> p3["Last 2 turns<br/>6K"]
    end
    subgraph after["AFTER — ~19,000 tokens (−92%)"]
        q1["System + tools<br/>8K"] --> q2["Structured state summary<br/>5K"] --> q3["Last 2 turns<br/>6K"]
    end
    style p2 fill:#fff7ed,stroke:#f97316,color:#7c2d12
    style q2 fill:#ecfdf5,stroke:#16a34a,color:#14532d,stroke-width:2px

• Preserved: System instructions, project memory files (e.g., CLAUDE.md or AGENTS.md), and the last $N$ raw turns so immediate conversational continuity feels seamless.
• Pruned & Replaced: Hundreds of thousands of tokens of historical tool calls, raw file dumps, and verbose command outputs are collapsed into a dense state block (typically 5K–8K tokens):

Notice the Ruled-Out Dead Ends field: explicitly recording what failed prevents the post-compaction agent from waking up and repeating debugging paths it already ruled out twenty minutes ago. Now, whether you return 2 minutes later or 6 hours later, your next message only has to prefill a lean ~19K-token context instead of 250K tokens.

4. Subtle Engineering Details in Production Harnesses

Building reliable idle compression requires more than setTimeout(). Four subtle edge cases separate toy implementations from production-grade harnesses:

1. Speculative Execution & Instant Preemption

Background compaction must never hold a blocking mutex over the user’s terminal. What happens if the timer fires at 266 seconds, compression is mid-stream at 269 seconds, and the user hits Enter with a new instruction?

flowchart LR
    A["Turn Ends<br/>(Context ≥ 40K)"] --> B["Idle Timer<br/>(Wait 266s)"]
    B -->|"User types < 266s"| C["Cancel Timer<br/>(Cache stays warm)"]
    B -->|"266s elapsed"| D["Speculative Compact<br/>(Stream summary)"]
    D -->|"User hits Enter!"| E["Abort Stream<br/>(Keep original context)"]
    D -->|"Summary finishes"| F["Swap to ~19K<br/>(Disarm timer)"]
    style D fill:#eef6ff,stroke:#3b82f6,color:#1e3a8a
    style E fill:#fff7ed,stroke:#f97316,color:#7c2d12
    style F fill:#ecfdf5,stroke:#16a34a,color:#14532d,stroke-width:2px

Well-designed harnesses treat idle compression as speculative work:

• Compaction runs on a cloned snapshot of the message array tied to an AbortController (or cancellation token).
• Any user prompt submission or interrupt signal immediately aborts the in-flight compaction HTTP stream, discards the partial summary, and dispatches the user’s message against the original, unmodified history—which just got its cache TTL refreshed anyway!

2. Minimum Token Thresholds & One-Shot Guards

If a conversation only contains 12K tokens, compacting it wastes output tokens for negligible savings. Similarly, once an idle session has been compacted down to ~19K tokens, the harness transitions into a CompactedIdle state and disarms the timer until substantial new tool output accumulates; otherwise, an unattended terminal tab would re-summarize its own summary every 4.5 minutes all weekend. Harnesses enforce a minimum compressible delta (e.g., only arm the timer if total context exceeds 40K–50K tokens since the last compaction).

3. Preserving Tool & Harness Invariants

Coding harnesses enforce safety invariants outside the raw LLM text—most notably read-before-edit tracking (verifying that the agent has read a file and that its modification timestamp hasn’t changed on disk before allowing an edit_file patch).

If idle compression blindly deletes the historical read_file tool-call messages without preserving sideband harness metadata (or re-injecting the most critical active working files), the agent will wake up knowing what code to change from its summary, try to call edit_file, and immediately hit a harness error complaining that it hasn’t read the file yet. Production harnesses decouple file-state bookkeeping from raw message pruning and instruct the compaction prompt to retain exact file paths, line numbers, pending TODO items, and unresolved test failures.

4. Multi-Tier TTL Awareness

As LLM providers introduce tiered caching (such as 5-minute default caches alongside opt-in 1-hour extended caches), hardcoded timers break down. Modern harnesses dynamically bind $T_{\text{trigger}}$ to the active model and cache-control configuration of the session.

5. Trade-offs, Failure Modes, and When (Not) to Use It

TTL-aware idle compression is not a free lunch. Because it trades exact historical recall and upfront speculative tokens for cheaper post-break resumes, enabling it blindly across every agent workload can hurt accuracy or even increase your API bill.

The Core Trade-offs

1. Lossy Memory & “Re-Read Churn” (Cost vs. Fidelity)

Summarization is inherently lossy. Collapsing 236K tokens into 5K inevitably discards exact line numbers, subtle stack-trace frames, and verbatim config values. If you step away during a deep root-cause investigation and return asking, “Wait—what was the exact error message in the second test shard?”, the agent no longer has the raw log in context. It must either re-read files and re-run commands (burning extra turns and tool latency) or risk hallucinating details from a compressed summary. (Mitigation: Production harnesses persist the raw JSONL transcript on disk so the agent can grep its own archived history on demand.)

2. The “Abandoned Tab Tax” (Speculative Spend on Dead Sessions)

Idle compression bets that an idle user will eventually come back. When a developer finishes their task, leaves the terminal open, and walks away for the day, doing nothing costs zero extra tokens—the KV cache simply evaporates at $t = 300\text{s}$. With idle compression, the harness wakes up at $t = 266\text{s}$, reads 250K cached tokens, and generates ~5K output tokens (which typically cost $4\times$–$5\times$ base input rates) for a session that never receives another prompt. Similarly, if a user frequently resumes right around $t \approx 268\text{s}$ and aborts an in-flight compaction stream, you still pay for the cached prefill and partial output tokens generated before cancellation.

3. Sensitivity to Provider Cache Economics (The Break-Even Math)

Whether speculative compaction saves money in expectation depends on your session resume probability $P(\text{resume})$ and the steepness of your provider’s cache cliff. Compacting costs $C_{\text{compact}}$ upfront to save $\Delta C_{\text{resume}}$ if the user returns after TTL expiry, breaking even only when:

$$P(\text{resume} \mid \text{idle at } T_{\text{trigger}}) > \frac{C_{\text{compact}}}{\Delta C_{\text{resume}}}$$

Look at how dramatically that break-even threshold shifts on our 250K-token baseline (generating a 5K summary to reach a ~19K compacted context):

On APIs with steep cache cliffs ($10\times$–$12.5\times$ gap between warm and cold turns), idle compression pays for itself even if > 80% of idle tabs are abandoned. On APIs with shallower $2\times$ cache discounts, unconditional idle compression loses money unless nearly two-thirds of idle sessions resume—making lazy post-TTL deterministic truncation a better default there.

4. Silent State Mutation & Auditability

Mutating prompt state on a wall-clock timer makes agent behavior non-deterministic across time: asking the exact same follow-up question at 4m 20s versus 4m 45s hits two materially different contexts. In regulated, forensic, or benchmark-evaluation environments, background lossy mutation complicates reproducibility unless explicitly logged and gated.

Decision Guide: When to Use It vs. When to Skip It

Key Takeaways for Agent Builders

As agents transition from single-shot chatbots into persistent background coworkers, context lifecycle management becomes systems engineering.

• Time matters as much as token count: Context management isn’t just about staying below max_context_length; it is about aligning memory footprint with KV-cache residency windows.
• Hide maintenance in human think-time: Moving compaction off the active turn and into natural user pauses makes long-horizon sessions feel instantaneous.
• Never break prefix symmetry: Appending your compaction instruction onto the live conversation tail turns an expensive summarization pass into a cheap, cache-warm operation.
• Model the expected-value trade-off: Idle compression trades exact recall and speculative output tokens for semantic state synthesis and cheap resumes—enable it where cache cliffs are steep and developers return after breaks, and prefer lazy post-TTL truncation or raw history for headless jobs, shallow-discount APIs, and precision-critical debugging.