Context Compression for Production AI Agents

From message trimming and tool-result cleanup to Hermes Agent’s layered compaction

Summary

As an AI agent repeatedly calls search, file, terminal, browser, and database tools, its context keeps growing. The problem is not only that the final request may no longer fit: every turn becomes more expensive and slower to first token; old tool output, repeated logs, and solved problems dilute useful information; and the model can forget constraints, repeat calls, or retry approaches that already failed. Blindly deleting history, however, breaks continuity in long-running work.

The goal of production context compression is not to produce the shortest possible summary. It is:

to preserve the state an agent needs to continue correctly, safely, and verifiably with as few active tokens as possible.

Using section 2.7 of Understanding AI Agents: Design Principles and Engineering Practice as a reading path, this article rechecks OpenAI, Anthropic, LangChain v1, Google ADK, Hermes Agent documentation, and current source code through July 2026. It distinguishes the book’s methodology, current products and frameworks, Hermes Agent’s behavior on its current main branch, and production-ready engineering advice.

1. Why context compression is necessary

1.1 The hard limit: context windows are finite

A typical agent loop continually appends user requests, model tool calls, tool results, analysis, further tool calls, and more results. The usual sources of overflow are not user messages but web pages or PDFs returning tens of thousands of tokens, large logs, many source files, browser snapshots and multimodal attachments, tool-call arguments containing full files, and historical reasoning or replay metadata.

After a model’s context window is exceeded, a request can fail outright. Remaining below the limit is not enough to make a session healthy.

1.2 Soft degradation: fitting is not the same as working well

LangChain v1 notes that even histories which still fit can be distracted by stale or irrelevant content while adding latency and cost. Anthropic calls the declining use of effective information as context grows context rot.^{1 2}

Context capacity: space remains
Information density: keeps falling
Model attention: is split by old logs, repeated results, and irrelevant detail

Compression therefore prevents both context_length_exceeded and a loss of decision quality.

1.3 Long-running agents need a handoff, not a chat summary

A chat summary such as “the user and assistant discussed a refund” cannot resume work. A long-running agent needs a checkpoint:

active_goal: Fix duplicate charges in the refund API
constraints:
  - Do not change the database schema
  - Preserve API backward compatibility
completed:
  - Reproduced duplicate charges under concurrent requests
  - Located payment/idempotency.py
  - Added idempotency-key validation
failed_attempts:
  - Redis TTL alone still has a race condition
verification:
  unit_tests: "47/50 passed"
  failing:
    - test_concurrent_refund
    - test_retry_after_timeout
    - test_duplicate_webhook
modified_files:
  - payment/idempotency.py
  - tests/test_refund.py
remaining:
  - Fix three concurrency tests
  - Run the full regression suite

It answers where the task is and how the next agent turn should continue, rather than merely what was discussed.

2. Four concepts that must not be conflated

Context compression / compaction

It changes the active context seen on the next model turn. Old content can be removed, externalized, or replaced with a summary.

Prompt cache

Caching does not necessarily reduce context length. A request may still contain 80,000 tokens while an identical 60,000-token prefix reuses prefill work. Caching avoids recalculation; compression avoids carrying the same material forever.

Long-term memory

Long-term memory preserves user preferences, business facts, or experience across sessions. It should not replace current task state or place all history permanently into the system prompt.

RAG and artifact retrieval

Keep complete originals outside the prompt and retain only a summary, an artifact_id, source information, and a way to retrieve the detail. This is the essential infrastructure for limiting compression loss.

3. The main context-compression approaches

Production systems normally combine several of the following, roughly from cheapest to most intelligent.

3.1 Message trimming and sliding windows

Keep only recent messages or a recent token budget; preserve the system prompt, optionally the original task, the latest real user request, and intact tool_call/tool_result pairs. LangChain v1 lists message trimming, permanent deletion, and summarization as basic short-term-memory strategies.¹

It is cheap, fast, deterministic, and free of summary hallucination, so it makes a good overflow safety net. Its information loss is high: early constraints and decisions can disappear, leading to repeated work. It is best for ordinary chat, simple independent turns, and fallback protection rather than as the sole long-task strategy.

3.2 Deterministic filtering and noise removal

Code can remove clearly low-value material: duplicate log lines, page chrome and advertising, empty results, streaming deltas already processed, repeated status notices, recomputable statistics, and expired progress messages.

def should_drop(message) -> bool:
    return (
        message.is_empty
        or message.is_duplicate
        or message.kind in {"heartbeat", "stream_delta"}
        or message.expired
    )

This has no LLM cost, is stable and testable, and should precede LLM summarization. It only handles recognized noise, rules need maintenance, and deleting material in the middle of a history can invalidate the prompt cache after it.

3.3 Tool-result cleanup and demotion

Keep the fact that a tool was called but replace a large body with a concise, retrievable record:

assistant:
  read_file("server.log")
tool:
  [read_file] server.log, 50,000 lines;
  found DatabaseTimeout 127 times and PermissionDenied 3 times;
  full content: artifact://logs/run-103

Anthropic’s Context Editing and Tool-result Clearing follow this pattern.² It is high-yield for tool-heavy agents and can avoid an LLM call while preserving the decision trace. It is only near-lossless when the tool remains callable, identical input returns equivalent output, retrieval is affordable, and the source remains available. Clearing real-time prices, one-time authorizations, temporary pages, or irreversible transaction responses may not be recoverable.

3.4 Externalize large content as artifacts

Avoid placing the full object into the main context in the first place. Store a 200 MB log in a file, object store, or database, and leave the prompt with its ID, type, size, structured statistics, a small preview, and available queries.

artifact_id: "artifact://pytest/run-20260726-103"
type: "test_log"
lines: 58321
summary:
  passed: 128
  failed: 2
failures:
  - test_refund_timeout
  - test_duplicate_webhook
available_operations:
  - grep
  - read_lines
  - download

OpenAI recommends putting large resources in a container filesystem or database so an agent can selectively open files and run queries rather than putting everything into the prompt.³ Artifacts work well for code, logs, browser pages, PDFs, database results, JSON, images, and screenshots, but require an artifact store, access control, lifecycle management, and reliable retrieval handles.

3.5 Rolling structured summaries

Preserve a stable head verbatim, compress the older middle trajectory into a checkpoint, and keep a recent raw tail:

System Prompt
+ Initial task / key constraints
+ Structured Checkpoint
+ Recent raw messages

LangChain v1’s SummarizationMiddleware, Google ADK Context Compaction, and Hermes’s default compressor use this family of designs.^{1 4 5} It greatly reduces context while retaining goals, decisions, progress, and unfinished work. It remains lossy, costs an LLM call, can omit numbers, paths, errors, and edge conditions, and repeated summarization can drift.

A checkpoint should contain fields such as historical_task, goal, constraints, completed_actions, active_state, blocked, key_decisions, resolved_questions, relevant_files, critical_context, remaining_work, and source_refs. “Some changes were made and a few problems remain” is not adequate.

3.6 Task-aware compression

Instead of merely asking a model to summarize history, specify the current task, the question now being answered, confirmed facts, missing facts, fields that must not change, and sources that must remain recoverable. A task-aware summary usually has greater information density, distinguishes known/unknown/conflicting items, and is valuable for research, due diligence, debugging, coding agents, and staged business processes. It can overfit a short-term goal: information removed for one task may matter again after a task switch.

3.7 Summaries with citations and traceability

Keep a source handle beside each material fact:

facts:
  - claim: "The service began timing out after 14:32"
    source_ids:
      - "artifact://logs/prod-103#L820-L912"
    confidence: high
    verified: true

The summary is lossy but the source index is as close to lossless as practical. This supports audits and retrieval in enterprise RAG, legal, finance, research, operations, and data analysis. It needs stable artifact IDs, permissions, and retention, and a citation alone does not make a summary correct.

3.8 Vendor-native compaction

OpenAI. The Responses API provides native compaction through responses.compact, used by the Agents SDK’s OpenAIResponsesCompactionSession. It can emit a type=compaction item containing opaque encrypted_content; Codex uses this mechanism for long-running coding work.^{3 6 7} It integrates closely with the provider and lowers implementation effort, but is hard to audit, creates lock-in, and may block streamed completion. For low-latency paths, manual compaction between turns or during idle time can be preferable.

Anthropic. Anthropic recommends server-side Compaction for long conversations, producing a replayable compaction block. Context Editing provides finer control over particular material such as tool results. These features are beta and must be checked against current documentation.^{8 2} They offer server-side token accounting, explicit message types, and customizable instructions, but remain lossy and can change. Replacing a default summary prompt requires the developer to supply a complete retention policy.

For rapid delivery within one vendor, evaluate native compaction first. For auditability, cross-vendor portability, and exact field control, keep a client-side structured checkpoint. Both can coexist: the client maintains canonical state while the provider manages its model trajectory.

3.9 Token-level prompt compression

LLMLingua, LongLLMLingua, and LLMLingua-2 delete or rearrange low-information tokens rather than writing a new summary:

original:   The system encountered a very serious database connection timeout.
compressed: system encountered serious database connection timeout

LLMLingua uses a small language model to estimate token importance; LLMLingua-2 frames compression as token classification.^{9 10} This can suit natural-language RAG passages, meeting notes, and recoverable background material. It can also remove negation, qualifiers, or exact numbers, damage JSON, code, SQL, and tool parameters, and impose compressor latency. A 2026 large-scale study found end-to-end acceleration only when input length, compression ratio, and hardware align; elsewhere compression overhead can erase the gain.¹¹

Do not compress tool-call JSON, structured tool-result fields, code, SQL, UUIDs, hashes, paths, ports, security policy, hard business constraints, money, dates, or legal terms without dedicated evaluation.

3.10 Sub-agent context isolation

This prevents noise from entering the main context at all. A search sub-agent may read many files and run grep, returning only the entry file, key functions, call sites, tests, and confidence. The main agent retains high-density conclusions. This is especially useful for code search and multi-source research, but requires complete task descriptions and introduces report omissions, extra model calls, and more complex scheduling, timeout, and error handling.

4. Comparison

5. Production practice: build a layered pipeline

Do not choose only one technique. Use this order:

Complete raw trajectory / Artifact Store
            │
            ▼
1. Admission Control: externalize large content and return structured previews
            ▼
2. Deterministic Cleanup: deduplicate, remove noise, clean refetchable tool results
            ▼
3. Canonical State Projection: code owns TODOs, counts, phases, and verification status
            ▼
4. Structured Compaction: turn the old middle into a task checkpoint
            ▼
5. Recent Raw Tail
            ▼
6. LLM

The model context must not be the only source of truth. Keep deterministic, validated canonical state in PostgreSQL, LangGraph State, Redis, or equivalent; retain full artifacts separately; and project a concise active context.

Use a cheap-first order: deduplication, explicit noise removal, large-object externalization, tool-result demotion, a state bar, LLM summary, then full compaction. Always keep a recent raw tail because summaries preserve conclusions better than the nuance, reference, and transient state of the latest interaction. Preserve complete tool-call/result groups.

Do not copy a fixed 50%, 70%, or 80% threshold. A safe trigger must account for the context window, reserved output, expected tool burst, safety margin, system prompt and schemas, multimodal input, and provider replay fields. Add hysteresis, a target post-compression level, minimum reclaim, cooldown, and an ineffective-compression circuit breaker.

Compression must be atomic: copy messages, compress the copy, validate the checkpoint and message structure, then commit once. On failure, preserve the original session, alert, cool down, or change summary models. Exact identifiers—paths, URLs, UUIDs, commit hashes, issue numbers, database IDs, money, dates, ports, error codes, test names, and versions—should be extracted by code or kept verbatim rather than entrusted to free-form summaries. Historical summaries should carry dates and task boundaries, and the newest real user message must remain authoritative.

6. Hermes Agent: a layered case study

6.1 Pluggable context engine

Hermes abstracts context management behind ContextEngine. Its default is the lossy ContextCompressor; plugins can explicitly select another engine such as LCM. This is preferable to embedding compression in the agent loop because models, audit requirements, and strategies differ.⁵

6.2 Two layers of triggering

Hermes documents a gateway session-hygiene safety net before agent processing at roughly 85% of context, plus an in-loop ContextCompressor whose documented default is 50% and uses actual API token counts. “Default 50%” is not universal behavior: current docs and source support per-model overrides, a raise-only 75% floor for windows below 512K, special Codex-route adjustments, and Codex app-server native thread compaction rather than rewriting Hermes’s local copy.^{5 12}

6.3 Deterministic pre-compression in current source

Before LLM summarization, current main deduplicates byte-identical tool results; rewrites old large results into tool-specific one-line summaries; preserves terminal commands, exit codes, line counts, file paths, read offsets, content size, search conditions, and hit counts; truncates huge tool-call parameters while keeping valid JSON; can demote completed large output even in the protected tail under severe pressure; replaces old images with placeholders; and inserts deterministic reload markers after trimming skill content.¹²

6.4 Head, middle, and tail

Hermes still preserves a verbatim head of system instructions and early anchors, summarizes the older middle trajectory, and protects a recent tail under token and minimum-message budgets. Boundaries try to keep tool calls and results paired.

6.5 Bounded summary input

Current main does not send an unlimited entire middle to a summary model. It truncates individual message bodies and tool-call arguments, caps the total summarization sequence at roughly 160,000 characters while retaining its beginning and end with an omission marker, targets a summary around 20% of compressed content, and currently caps summary output at 10,000 tokens rather than the older 12,000.¹²

6.6 A handoff format and protection from reviving old tasks

The structured snapshot includes Goal, Constraints & Preferences, Completed Actions, Active State, Blocked, Key Decisions, Resolved Questions, Relevant Files, Critical Context, and Pruned Skills, with concrete paths, commands, results, tests, errors, decisions, environment state, and skill reload instructions.¹²

Hermes marks the snapshot REFERENCE ONLY: the latest real user message is the sole authority for current work, completed historical requests must not be rerun, and new stop, undo, or never mind signals supersede old plans. Deterministic task anchors are extracted from real user messages rather than invented from examples or summaries.

6.7 Rolling updates, safety, and trade-offs

Later compaction combines the old summary with newly completed work, errors, and verification so active work moves to completed and stale state is updated. Current source includes boundary redaction, URL-credential and secret cleanup, removal of inline thinking, empty-summary detection, abort handling for authentication/network and selected failure paths, cooldowns, fallback counters, anti-thrashing, real-token post-checks, and tool-call/result orphan cleanup.¹²

Its strengths are cheap-first processing, targeted tool-result demotion, a preserved recent trajectory, structured handoffs, cache awareness, per-model thresholds, replaceable compressors, and mature failure paths. Its risks remain summary loss and drift, fast-changing source ahead of prose documentation, weak auxiliary models, large static system/tool prefixes, and cache-thrashing from thresholds that are too low.

7. Applying the design in LangChain v1 / LangGraph v1

7.1 Simple case: SummarizationMiddleware

from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
from langgraph.checkpoint.memory import InMemorySaver

agent = create_agent(
    model="gpt-5.4",
    tools=[],
    middleware=[
        SummarizationMiddleware(
            model="gpt-5.4-mini",
            trigger=("tokens", 40_000),
            keep=("messages", 20),
        )
    ],
    checkpointer=InMemorySaver(),
)

This is suitable for a prototype or ordinary chat. Production should use persistent checkpoint storage such as PostgreSQL rather than in-memory state.¹

7.2 Tool-heavy agents: layered custom middleware

from typing import TypedDict


class AgentState(TypedDict):
    messages: list
    active_goal: str
    constraints: list[str]
    todo: list[dict]
    tool_counts: dict[str, int]
    modified_files: list[str]
    verification: dict
    artifact_refs: list[dict]
    compaction_version: int

from copy import deepcopy


def compact_context(messages, state, policy):
    original = messages
    working = deepcopy(messages)  # work on a copy for atomicity
    working = deduplicate_tool_results(working)
    working = externalize_large_payloads(working)
    working = demote_old_refetchable_tool_results(working)
    working = strip_old_images(working)

    if estimate_tokens(working) < policy.trigger_tokens:
        return working

    head, middle, tail = split_head_middle_tail(
        working,
        keep_recent_messages=policy.keep_recent_messages,
        keep_recent_tokens=policy.keep_recent_tokens,
        preserve_tool_pairs=True,
    )
    try:
        checkpoint = summarize_middle(
            middle=middle,
            canonical_state=state,
            required_exact_fields={
                "paths", "ids", "hashes", "errors", "test_names", "amounts", "dates",
            },
        )
        validate_checkpoint(checkpoint, state)
    except Exception:
        return original

    compacted = sanitize_tool_pairs(head + [checkpoint.to_message()] + tail)
    return compacted if estimate_tokens(compacted) < estimate_tokens(original) else original

Keep state and summary separate. Code should update trusted facts such as tool_counts, verification totals, and artifact sources; the summary model should organize them with unstructured history for the next model turn.

8. Choosing a trigger threshold

Use a budget rather than a copied percentage:

context window
- maximum expected output
- next-turn maximum tool result
- system prompt and tool definitions
- multimodal budget
- safety margin
= latest safe input level

Monitor real and estimated prompt tokens, static-prefix tokens, recent-tail tokens, summary tokens, tool-result tokens, and artifactized tokens. Configure trigger_tokens, target_tokens_after_compaction, minimum_reclaim_tokens, cooldown_seconds, max_compactions_per_task, and ineffective_compaction_limit.

9. How to evaluate a compression system

Do not test compression ratio alone. Measure task completion, next-action consistency, retention of the original goal, safety and business constraints, unfinished TODOs, failed approaches, exact IDs/paths/hashes, citation traceability, duplicate tool calls, and drift after repeated compactions. Measure input tokens, compaction cost, first-token and end-to-end latency, prompt-cache changes, artifact refetches, compactions per 100 turns, ineffective compaction, and failure/fallback ratios.

Test timeouts, empty summaries, 429s and quota exhaustion, authentication failures, still-over-limit output, split tool pairs, a latest “stop” message conflicting with an old plan, secrets in summaries, rewritten commit hashes after repeated summaries, unavailable cleared results, and repeated Base64 images.

10. Selection guide

Ordinary chat: recent-message trimming + lightweight rolling summary + session database
Coding agent: artifactized files/logs + tool-result demotion + structured checkpoint
              + programmatic paths/hashes + isolated search sub-agents
Deep research: externalized results + task-aware cited summary + retrieval isolation
                + native compaction where useful
High-risk business: canonical-state database + event log + auditable structured summary
                    + tool-result references + code-level hard constraints
Low-latency real time: deterministic cleanup first + idle-time compaction
                       + small recent tail

11. Corrections to an earlier version

The current evidence requires several qualifications: Hermes’s 50% setting is not a universal trigger; it no longer merely replaces old output with a generic placeholder; current source bounds summary input and uses a 10K rather than 12K output ceiling; failure is no longer accurately described as silently deleting the middle; tool-result clearing is not inherently lossless; native compaction differs by vendor in opacity; experimental prompt-compression speedups are not production promises; and Google ADK currently supports both token and event-window strategies.

12. Final recommendation

1. Keep complete originals outside the prompt.
2. Artifactize large tool output at admission.
3. Use code for deduplication, noise removal, and canonical state.
4. Demote old, refetchable tool results.
5. Create a structured task checkpoint at a measured threshold.
6. Preserve a recent raw tail.
7. Give summaries source handles and protect exact identifiers.
8. Preserve the original session if compression fails.
9. Use hysteresis, cooldowns, and an ineffective-compression circuit breaker.
10. Evaluate task success, not compression ratio alone.

Do not make the model carry a whole ledger. Give it a trustworthy current state, a retrievable evidence index, and a sufficiently complete recent scene.

References

Checked on 2026-07-26. Agent APIs, models, and frameworks change quickly; verify vendor documentation and source for the pinned deployment version before production use.