A model can fail a test because it cannot solve the problem. It can also fail because the harness keeps making it forget what it already learned.
OpenAI’s July 29 ARC-AGI-3 report makes the second possibility unusually measurable. With the benchmark’s official harness, GPT-5.6 Sol scored 13.3% on the public task set. In a Responses API harness that retained prior reasoning and compacted long context, it scored 38.3% while using six times fewer output tokens. The model did not change between those two headline conditions. The memory policy around it did.
That result is OpenAI’s experiment, not an independent replication by TheGitReporter. But the production claim is inspectable. The open-source Codex repository contains the machinery that requests opaque reasoning state, records it, carries it into later samples, watches the context budget, and replaces an overgrown history with a compacted continuation. A sweep of the other projects tracked by this publication found close implementations in Pydantic AI, Pi, Hermes Agent, Mistral Vibe, and OpenCode; partial analogues in Qwen Code, Gemini CLI, and Cline; and compaction without equivalent OpenAI reasoning replay in Goose’s inspected Responses path.
The finding changes the useful question. It is no longer, “Which projects have a summarize button?” It is, “Which exact state reaches the next model call, and what happens to it when the window fills?”
The two settings are two different kinds of memory
Retained reasoning carries model-generated reasoning items from earlier calls into a later sample. This does not mean a harness reads or exposes raw private chain-of-thought. OpenAI’s current reasoning guide describes opaque reasoning items: a stored conversation can be continued with a prior response ID, while a stateless client can replay encrypted reasoning content. For models that support it, reasoning.context: all_turns makes compatible earlier items available to the next sample.
Compaction solves a later problem. Eventually the full transcript, tool results, observations, and reasoning state will not fit. Rolling truncation simply drops old material. Compaction produces a smaller state representation intended to preserve what the next turn needs. OpenAI’s compaction documentation now describes both automatic server-side compaction and a standalone compact operation; the returned compacted item is opaque and can carry prior state and reasoning into the next window.
Those mechanisms are complementary, not interchangeable. Retention prevents the harness from discarding useful reasoning after every action. Compaction prevents the longer-lived record from eventually becoming too large. A conventional prose summary can help with the second problem while doing nothing about the first.
| ARC-AGI-3 condition | Prior private reasoning | Old actions near the limit | Public-set RHAE |
|---|---|---|---|
| Official harness | Discarded after each action | Oldest messages dropped after a 175,000-character window | 13.3% |
| OpenAI Responses harness | Retained across turns | Compacted at a 175,000-token limit | 38.3%, with 6× fewer output tokens |
The table reports OpenAI’s published comparison. It does not isolate the contribution of each setting, publish uncertainty across repeated runs, or establish that another model or task will receive the same gain. That missing ablation is precisely what open harness developers can now make legible.
Codex turns the settings into a production chain
The cleanest source trail begins in Codex’s core/src/client.rs. A Responses request asks for reasoning.encrypted_content. On an eligible WebSocket continuation, the client checks that the next request is an incremental extension of the previous one and that relevant properties still match; only then does it send the new items with previous_response_id. If that continuity check fails, Codex can send the fuller request instead of pretending an unsafe prefix is reusable.
The response is not thrown away after text is rendered. Completed response items—including reasoning items—flow through stream_events_utils.rs into the session’s conversation history. The session recorder appends those raw response items and persists the rollout. That is the second link in the chain: requesting encrypted reasoning is useless if the harness drops the resulting item before the next call.
Then comes the context boundary. Codex checks its active window before sampling and again during a turn. The turn loop can trigger automatic compaction before a new sample, after tool work has grown the same turn, or before a model change would make the existing history too large. The context-window policy decides whether the configured limit applies to the total window or the body after a stable prefix.
The implementation has more than one compactor because “production” includes provider and version boundaries. The turn loop can select a newer streamed remote path, the older /responses/compact endpoint, or local summarization when remote compaction is unavailable. In the newer remote-v2 path, a streamed ResponseItem::Compaction becomes the replacement history alongside retained user, developer, and system messages, with a 64,000-token budget for those retained messages.
So the production technique is not a pair of checkboxes. It is a chain with at least five obligations:
- Ask the provider to return reusable reasoning state.
- Persist that state without exposing it as ordinary text.
- Replay it only when the provider, model, endpoint, and history shape remain compatible.
- Compact before the window becomes an emergency.
- Recover when a remote compactor, continuation pointer, or encrypted item cannot be reused.
Break any link and “reasoning retention enabled” can become a misleading configuration label.
Which tracked harnesses carry the same memory?
The comparison below is deliberately strict. “Opaque replay” means the inspected source carries a provider-issued reasoning item, encrypted payload, signed thinking block, or server-side continuation—not merely a human-readable assistant summary. “Compaction” names the concrete long-context policy, not a claim of equal quality. No project in this table has publicly reproduced OpenAI’s 38.3% result in the evidence reviewed for this article.
| Project | Reasoning continuity found in source | Long-context behavior | Closest practical reading |
|---|---|---|---|
| Codex | Encrypted reasoning items plus guarded previous_response_id continuation |
Automatic pre-turn and mid-turn remote or local compaction | The production reference implementation, with several compatibility and fallback paths |
| Pydantic AI Harness | all_turns, automatic prior-response chaining, encrypted signatures, and item IDs |
OpenAI native compact capability plus tiered model-agnostic trimming and summarization | The most explicit reusable toolkit for assembling the same policy outside one coding product |
| Pi | Encrypted reasoning replay and WebSocket prior-response continuation in its OpenAI adapters | Local iterative summary; reserves 16,384 tokens and keeps about 20,000 recent tokens by default | Both halves exist, but Pi’s compactor is provider-agnostic rather than OpenAI-native |
| Hermes Agent | Stores and replays codex_reasoning_items; detects endpoint-bound encrypted-content failures and retries without stale blobs |
A mature iterative context compressor with protected head/tail state, tool pruning, cooldowns, and telemetry | Strongest evidence that replay failure handling matters as much as replay support |
| Mistral Vibe | Collects OpenAI encrypted reasoning into reasoning_state and emits it on later requests |
Automatic model-based summary compaction, overflow retry, a configurable threshold, and failure telemetry | A compact, test-covered two-half implementation using local summaries |
| OpenCode | Stored item references or encrypted reasoning replay when store: false |
A dedicated compaction agent carries the prior summary and recent tail; media overflow has a separate recovery path | Close in mechanism, but reasoning metadata can be downgraded when a session crosses model families |
| Qwen Code | Provider-specific reasoning_content, Anthropic signed thinking, and Gemini thought signatures |
Cheap microcompaction of stale tool/media output before a full summary; 85% proportional trigger plus absolute headroom | Excellent context hygiene, but not one provider-independent equivalent of OpenAI all_turns |
| Gemini CLI | Gemini thought signatures preserve tool-loop continuity; the advanced manager can retain or distill thought nodes | Default profile masks large tools, degrades blobs, distills nodes, and builds state snapshots; legacy compression summarizes older history and verifies the summary with a second pass | A different provider’s continuity contract paired with unusually layered context management |
| Cline | Its message layer stores provider reasoning details and signed thinking metadata | Optional basic or agentic compaction at 90% of usable input, preserving 20,000 recent tokens by default | The machinery exists, but VS Code auto-condense is off by default in the inspected state |
| Goose | No encrypted OpenAI reasoning replay or prior-response chaining found in the inspected Responses formatter | Structured summary at 80% by default; progressively removes middle tool responses if the summary request itself is too large | Strong semantic compaction, but only one of OpenAI’s two demonstrated levers in the inspected path |
The source receipts behind that table are concrete. Pydantic AI’s OpenAI model adapter exposes prior-response continuation, all_turns, encrypted signatures, and the compact endpoint, while its compaction guide routes developers between native and tiered model-agnostic strategies. Pi’s Codex Responses adapter carries the prior response ID and asks for encrypted reasoning, while its compactor preserves a recent tail and accumulates file-operation state across summaries.
Hermes Agent’s Responses adapter goes further on a nasty production edge: an encrypted item is sealed to the endpoint that issued it, so Hermes stamps issuer identity, avoids replay across incompatible endpoints, and has a retry path for invalid_encrypted_content. Its context compressor separately owns lossy history reduction. Mistral Vibe’s OpenAI backend converts encrypted reasoning items into replayable state; its agent loop triggers compaction and records failures.
OpenCode’s new native Responses protocol chooses item references for stored responses and full encrypted reasoning items for stateless replay. Its session compactor is still a local agent-generated handoff. Qwen Code’s microcompactor first blanks stale, repeatable tool output and tells the file cache what must be read again; its full compressor reserves summary headroom and stops retrying after repeated failures.
Gemini CLI’s default context-management profile uses immediate sanitization, node distillation, truncation, and asynchronous state snapshots; its older chat compressor preserves a recent 30% and asks a second model call to find omissions in the summary. Cline’s SDK compaction pipeline can run a cheap basic strategy or an agentic one with fallback. Goose’s context manager makes old messages user-visible but agent-invisible, inserts a structured summary, and yields entirely when a provider says it manages its own context.
One tracked boundary remains intentionally unscored. The current OpenHands repository is the Agent Canvas frontend, not the full agent-server implementation. It exposes and tests a condenser configuration passed to the server, but the algorithm behind that server setting is not present in the inspected repository. A settings form is evidence of a capability boundary, not evidence of how memory survives inside it.
A fair way to test the harness instead of admiring it
Every team above can turn OpenAI’s result into a useful open experiment without copying ARC-AGI-3. Run the same long-horizon task in a two-by-two design:
| Condition | Earlier reasoning available? | History policy at the limit | What it isolates |
|---|---|---|---|
| A: forgetful baseline | No | Rolling truncation | The generic-harness failure mode |
| B: reasoning only | Yes | Rolling truncation | Continuity before the window fills |
| C: compaction only | No | Compaction | Long-history preservation without prior private reasoning |
| D: both | Yes | Compaction | The combined production-style policy |
Hold the model, prompt, tools, action budget, context limit, and task set constant. Record task success, output and input tokens, wall time, compaction calls, retries, and failure reasons. For a coding task, add state-recovery checks: can the agent still name the current hypothesis, the files it modified, the test that failed, and the next action after compaction? Publish enough repeated runs to show variance rather than one lucky trajectory.
Do not log private reasoning text to make the experiment observable. Log whether a compatible opaque item or continuation ID existed, whether it was accepted, which context mode was effective, and when compaction replaced the prior window. The object under evaluation is the state-transfer contract, not the contents of private thought.
Questions worth taking to each maintainer
The source makes the next developer conversations specific. These are not requests to chase one benchmark number; they are requests for ablations that expose what each harness already does differently.
- Codex: can the team publish a coding-task two-by-two that separates
all_turnsreasoning from remote and local compaction, including fallback rates and output-token cost? Ask the Codex maintainers. - Pydantic AI: can the Harness ship a named reference policy that combines automatic prior-response chaining, reasoning context, native compaction, and tiered fallback—then expose that policy as a reproducible eval fixture? Ask the Pydantic AI maintainers.
- Pi: does OpenAI-native compaction outperform Pi’s local summary on long coding sessions, and can native compacted state coexist with Pi’s cumulative file and branch metadata? Ask the Pi maintainers.
- Hermes Agent: how often does endpoint-bound encrypted replay fail during real provider failover, and what success or token penalty follows the safe retry without that state? Ask the Hermes maintainers.
- Mistral Vibe: can its existing compaction telemetry expose the four conditions directly, including empty-summary, tool-call, overflow, and replay failures? Ask the Vibe maintainers.
- OpenCode: how do item references, encrypted stateless replay, and a prior-response chain compare on cost and reliability—and how much state is lost when reasoning metadata crosses model families? Ask the OpenCode maintainers.
- Qwen Code: can the team ablate provider reasoning replay, microcompaction, and full summarization separately, while checking that evicted file reads are actually recovered? Ask the Qwen Code maintainers.
- Gemini CLI: what is the measured contribution of thought signatures, background state snapshots, and the second-pass summary probe—and which task states still disappear? Ask the Gemini CLI maintainers.
- Cline: why is auto-condense off by default in the inspected VS Code state, and what quality, latency, and privacy result would justify enabling basic or agentic compaction by default? Ask the Cline maintainers.
- Goose: can the OpenAI Responses provider add opaque reasoning replay and compare it with the current structured-summary-only path without double-compacting providers that own context? Ask the Goose maintainers.
- OpenHands: where can developers inspect the server-side condenser’s retention rules, and can the project publish condenser-on/off trajectories through the same Canvas-visible event stream? Ask the OpenHands maintainers.
The benchmark result is a warning about hidden experimental variables
OpenAI’s result does not prove that every model deserves a custom harness until it wins. A provider-specific harness can make comparisons less portable, and a model-authored summary can preserve the wrong facts with great confidence. Native compaction can add latency or cost. Encrypted reasoning can become invalid across endpoints. A tool result removed as “repeatable” may no longer be repeatable after the workspace changes.
But generic does not mean neutral. Discarding one model’s trained continuity state while preserving another provider’s signed thought blocks is a policy choice. Truncating the oldest actions instead of compacting them is a policy choice. Comparing two agents after one kept its file ledger and the other lost it is a policy choice.
The most honest next generation of agent leaderboards should publish a harness memory card beside the score: reasoning continuity mode, state-storage mode, compaction trigger, retained-tail budget, tool-output pruning, failure fallback, and model-switch behavior. Until then, the model is not the only thing taking the test. The invisible memory system around it is taking the test too.