OpenAI's Own Guide Puts 25 Benchmark Points in the Harness

This week, the agent harness stopped being plumbing and became a line item: DeepSeek open sourced one, Writer sells one, and NVIDIA routes through one. Four independent benchmarks published in the same window show why: swap the harness under a fixed model and the score moves 20 to 40 points, with almost no correlation between how models rank under one harness versus another.

The Score Was Never Just the Model

OpenAI’s own developer guide for GPT-5.6 makes this argument first, and makes it about OpenAI’s own model. On ARC-AGI-3, GPT-5.6 Sol scored 13.3 percent using the standard harness. Turning on two API level settings, persisted reasoning across turns and native context compaction, without changing the model at all, pushed that score to 38.3 percent, using roughly six times fewer output tokens. OpenAI’s own summary is blunt about it: “No changes to the model, but nearly three times the performance.”

That number deserves a second look, because it is the one figure here that comes from a vendor grading its own model, and it happens to hand that vendor’s own thesis its cleanest proof. OpenAI is telling its developers, in its own documentation, that 25 points of benchmark score live in the harness configuration and never touched the checkpoint.

ARC-AGI-3 score comparison under standard versus persisted reasoning and native compaction harness

Four Benchmarks, One Pattern

The ARC-AGI-3 jump would be a curiosity if it stood alone. It does not. Four more benchmarks published in the same week, independent of OpenAI and of each other, describe the same regularity from four different angles.

On SWE-bench Pro, changing the harness alone moved GLM-5.2’s score from 23 percent to 52 percent and Gemma 4 26B’s from 15 percent to 36 percent, according to an analysis circulated in Latent Space’s AI News roundup. Across the harnesses compared, the rank correlation between how models placed under one harness versus another came out to negative 0.05: a harness leaderboard carries essentially no information about how the same models would rank under a different harness.

On PostTrainBench, a benchmark that measures how well an AI system can improve an open-weight model’s own performance, the startup Intology’s Locus system lifted Claude Opus 5 from 34.1 percent to 44.7 percent by wrapping it in a dedicated research harness, according to Import AI. On the extended PostTrainBench+ variant, which allows far more compute, Locus reached 51.6 percent, above the benchmark’s human baseline of 51.1 percent.

On OfficeQA Pro V2, a Databricks financial-analysis benchmark, GPT-5.6 Sol scored 33 percent running on its own inside Codex. Adding Mixedbread’s Toast 1 as a dedicated search subagent, with the underlying model unchanged, brought that to 70 percent at roughly 1.15 dollars per task, the highest score Databricks recorded across every system it evaluated in that release.

And on AppWorld, IBM Research’s ALTK-Evolve memory system held DeepSeek-V3.2 fixed and compared it against a rival memory system called ACE. ALTK-Evolve completed 89.3 percent of tasks against ACE’s 80.4 percent, while spending 263,000 tokens per task against ACE’s 634,000.

Four benchmarks, four teams, four unrelated tasks. None of them changed the model. All of them moved the score by double digits just by changing what surrounds it.

Score change at constant model across four benchmarks: SWE-bench, PostTrainBench, OfficeQA, AppWorld

The Harness Ships as a Product This Week

This pattern would be a research footnote if the harness were still internal tooling. It is not. Three companies shipped one this week, each treating it as a commercial layer distinct from the model running inside it.

DeepSeek released DeepSeek Harness, an open-source agent runtime under the MIT license, positioning it explicitly as a rival to Anthropic’s Claude Code. The project’s architecture follows a single principle DeepSeek states directly: everything is a plugin, meaning the model, tools, sandboxes, and orchestration logic can each be swapped independently. The repository drew roughly 27,500 GitHub stars and 2,000 forks on launch day, a snapshot rather than an adoption figure.

Writer took the opposite packaging decision and sold a harness alongside a model it did not train from scratch. Palmyra X6 is a post-trained version of Z.ai’s open-weight GLM-5.2, built on just 626 curated training trajectories rather than a fresh pretraining run. Writer published two overlapping but distinct sets of savings numbers, and they should not be collapsed into one. According to TechCrunch, harness changes alone cut costs by an average of 40 percent across the models Writer tested internally, a more reliable lever, the researchers wrote, than switching models at all; the model and harness combined were estimated to cut costs by up to 50 percent on basic tasks. According to a separate VentureBeat breakdown, the rebuilt Writer Agent harness alone cut costs 41 percent and completed tasks 44 percent faster across every model tested, including third-party models from Anthropic and OpenAI; paired specifically with Palmyra X6, the combined system delivered a 52 percent cost reduction, a 48 percent speed improvement, and a 10 percent quality gain on Writer’s internal evaluations. Two publications, two slightly different numbers, one consistent shape: the harness moves the needle on its own, and stacking a purpose-built model on top of it moves it further.

NVIDIA shipped NeMo Switchyard the same week, an open-source library that routes each step of an agent workflow to whichever model in a mix, open, proprietary, or NVIDIA’s own, fits that step best. NVIDIA’s internal benchmarks put the resulting cost per completed task at roughly a third of running Claude Opus 4.8 for everything. Partners reported their own numbers on top of that: LangChain cut cost 74 percent across 145 multi-turn Deep Agents tasks by routing only 7 percent of calls to a frontier model, accepting a 6 percent accuracy tradeoff; Ramp cut cost 58 percent and runtime 33 percent on its SWE-Bench suite while matching frontier-model performance; Cognition cut mean cost 28 percent inside Devin Desktop, in a deployment built for NVIDIA’s own internal use, while holding near-frontier accuracy. Every one of those figures is a vendor or partner measurement, not an independent benchmark.

Comparison of agent stack harness packaging across DeepSeek, Writer, and NVIDIA

Two Different Kinds of Cheap

Not all cheap is the same cheap, and this week’s clearest illustration comes from outside any company’s benchmark suite. A solo entrant in a GPU Mode contest steered Codex running GPT-5.5, with Claude consulted along the way, through more than 1,500 submissions over 14 days to optimize a CUDA kernel for QR decomposition. The baseline solution ran in roughly 419,000 microseconds; the final tracked result ran in 1,805 microseconds, a 232 times speedup that placed the entrant 12th out of 183. The tools that got there were harness techniques, not model upgrades: a /goal mode that let the model loop unsupervised for a day at a stretch, a beam of three to five candidate solutions kept alive at once so a single bad early result did not kill a promising direction, and a stronger advisor model consulted mid-session for fresh ideas whenever the main agent stalled.

None of that shows up in a price-per-token comparison, and that gap is exactly where the industry keeps blurring two different numbers. Anthropic’s own guidance on running Claude Code sessions lays out why: output tokens are priced at roughly five times the rate of input tokens, because generating a response keeps the model running one token at a time, while reading from the prompt cache costs a tenth of the input price and writing to it costs up to twice as much. A cheaper model that needs more turns, more retries, more context reread on every step, can lose its per-token advantage entirely once the loop is counted end to end. Price per token measures the rate card. Cost per completed task measures the harness wrapped around it.

Claude Code session pricing multipliers relative to standard input tokens

If the Harness Decides the Score, Can the Score Judge Itself?

If the harness decides the score, the obvious next question is whether the people scoring the harness can be trusted either. This summer, Hugging Face ran exactly that test on a different kind of benchmark: peer-reviewed science itself.

The ICML 2026 Open Reproductions challenge asked 1,221 participants, working with coding agents including Claude Code, Codex, and Cursor, to reproduce the claims in roughly 2,200 papers accepted to the conference, using $20 in cloud compute credits each and 2,962 cloud jobs in total. Fifty one percent of the papers examined, 1,103 of them, had at least one claim independently verified, with 266 fully reproduced. Twenty three percent, 496 papers, had at least one claim falsified or contested. Within that group, 242 papers saw two independent reproduction teams reach opposite verdicts on the identical claim.

Outcome breakdown of the ICML 2026 Open Reproductions challenge

That is a different exercise from the NeurIPS shadow evaluation covered previously in our piece on the cost of verification, where the same papers’ own authors rated an identical Opus 4.8 judgment a 2 in one instance and a 1 in another. Different conference, different papers, different protocol; both point at the same open problem, that judgment, whether of code or of a scientific claim, is not yet solved by adding an agent to the process.

For an argument built entirely on benchmark numbers, that is the honest complication to sit with rather than explain away. The four benchmarks in the earlier section were each run once, by one team, and reported as a result. ICML’s own reproduction effort suggests that roughly a quarter of published claims would not survive a second independent look, and that two independent teams checking the identical claim agree with each other only most of the time, not always.

What It Means

Four implications follow from putting this week’s numbers side by side.

The harness now belongs on the same line as the model name on any deployment’s technical specification. A team that reports which checkpoint it is running without reporting the harness configuration around it is reporting half of what actually produced the result.

A benchmark score published without its harness configuration is close to unusable for comparison. SWE-bench Pro’s rank correlation of negative 0.05 is the sharpest statement of this: a leaderboard built on one harness says almost nothing about how the same models would place on another.

Harness engineering is becoming a budgeted discipline in its own right, not a footnote to which model a team picks. DeepSeek, Writer, and NVIDIA are not competing on parameter counts this week; they are competing on what wraps around the model once it is running, and each picked a different way to sell that layer.

And the measurement layer itself, evals and now reproductions of published research, deserves the same scrutiny the industry already applies to what an agent does in production. If a harness can move a score 25 to 40 points without touching the model, the question of who checked the harness, and who checked the checker, is no longer a side note.