# Same model, fewer round trips: graff against the Codex app server and Claude Code

Same model, same 21 tasks: graff finished 22% faster than OpenAI's Codex app server and 34% faster than Claude Code, mostly on MCP work, where it makes fewer round trips and keeps less in context. Claude Code and pi are still quicker on small coding tasks.

Published: 2026-10-01
Author: Rach Pradhan
Canonical: https://codegraff.com/blog/graff-vs-codex-and-claude-code

A coding agent is two things: the model, and the harness around it that decides what the model sees, which tools it can call and how many round trips a task takes. Hold the model fixed and whatever difference you measure is the harness. So we gave graff, OpenAI's Codex app server, Claude Code, OpenCode 2 and pi the same models and the same 21 tasks, three runs each. graff finished in 22% less time than the Codex app server and 34% less than Claude Code, almost all of it on MCP work. On small coding tasks, Claude Code and pi are still a little quicker.

![Mouse engineers compare differently shaped brass coding machines that receive matching task cards.](https://codegraff.com/blog/graff-frontier-harness-evals/harness-exhibition.webp)

Different machines, the same task cards. Conceptual illustration.

## The short version

We ran 21 tasks: 11 small coding jobs (fix a bug, rename a function, write tests, transform JSON) and 10 jobs that read a Linear-shaped MCP server, fetching issues and their comments and writing a report. Every run had to pass the task's own check, and every harness passed every run.

There are two comparisons, because no one model runs in all five harnesses: graff against the Codex app server on gpt-6.1-sol with one ChatGPT account, and graff against Claude Code, OpenCode 2 and pi on one model served through the Codegraff gateway.

Sources: [codex-results: graff vs the Codex app server and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/RESULTS.md); [cc-results: graff vs Claude Code, OpenCode 2 and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-claude-code/RESULTS.md)

## Against the Codex app server

OpenAI's Codex app server is the engine inside the Codex apps. We drove it through its JSON-RPC interface on gpt-6.1-sol, with the same ChatGPT account graff used, both at the model's default effort.

graff took 22.2 seconds per task against 28.6, and the gap is about the same on both kinds of work: 16.0 against 20.4 seconds on the coding tasks, 29.0 against 37.8 on the MCP tasks. graff made 3.1 model calls per task to the app server's 4.4, and 3.3 to 5.8 on the MCP tasks, where one rlm script often fetches, trims and writes the report in a single step.

It cost less too: $0.013 per task against $0.043 at list price, 69% less. That is a floor for the app server, which also sends one warm-up request per thread that its usage does not report.

pi, a minimal open-source harness, has no MCP client, so it ran the coding tasks only: 17.0 seconds per task against graff's 16.0. Its prompt is about a tenth the size of graff's, so it costs less there.

gpt-6.1-sol, one ChatGPT account. Mean per task over 3 runs; list cost at the model's public rates.

| Harness | All tasks | Coding | MCP | Calls per task | List cost per task |
| --- | --- | --- | --- | --- | --- |
| graff | 22.2s | 16.0s | 29.0s | 3.1 | $0.013 |
| Codex app server | 28.6s | 20.4s | 37.8s | 4.4 | $0.043 |
| pi | — | 17.0s | no MCP client | 3.2 (coding) | $0.007 (coding) |

Sources: [codex-results: graff vs the Codex app server and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/RESULTS.md)

## Against Claude Code, OpenCode 2 and pi

Claude Code, OpenCode 2 and pi ran on one model served through the Codegraff gateway, with the same model id and key for every harness. Claude Code speaks only Anthropic's Messages API and the gateway takes OpenAI-style chat completions, so Claude Code reached it through a local LiteLLM proxy.

graff took 8.7 seconds per task, Claude Code 13.2 and OpenCode 2 14.1. On the MCP tasks graff took 10.6 seconds against 21.2 and 22.1: half the time.

On the coding tasks the order flips. Claude Code and pi took 6.0 seconds per task, OpenCode 2 6.8 and graff 7.0, and about 0.3 seconds of graff's is one slow gateway response on a one-call task. Small tasks on this model are where graff has the least to gain.

One gateway model, same id and key for every harness. Mean per task over 3 runs.

| Harness | All tasks | Coding | MCP | Calls per task | Prompt tokens per MCP task |
| --- | --- | --- | --- | --- | --- |
| graff | 8.7s | 7.0s | 10.6s | 3.5 | 44k |
| Claude Code | 13.2s | 6.0s | 21.2s | 3.5 | 179k |
| OpenCode 2 | 14.1s | 6.8s | 22.1s | 4.3 | 63k |
| pi | — | 6.0s | no MCP client | 2.8 (coding) | — |

Sources: [cc-results: graff vs Claude Code, OpenCode 2 and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-claude-code/RESULTS.md)

## Where the time goes

The MCP gap is mostly context. Claude Code keeps every MCP result whole, so each call re-sends everything fetched so far: 179k prompt tokens per MCP task against graff's 44k, about 40k per call against 10k. It also wrote nearly twice the output, 1,645 tokens per MCP task against 882.

graff trims large list results before they reach the model, and often runs the fetch, filter and write loop as one rlm script, so the model reads less and is called fewer times. OpenCode 2 sits in between at 63k prompt tokens per MCP task, but made the most calls, 5.5 per task.

Task by task, graff was the fastest of the three on all 10 MCP tasks.

Sources: [cc-results: graff vs Claude Code, OpenCode 2 and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-claude-code/RESULTS.md)

## What changed in graff

Each of these removes a model call that did no work the task asked for, and every call costs seconds.

Servers you name load up front. When a prompt names an MCP server ("the linear server"), graff loads its tools with the first request instead of spending a round trip on a load call, and asking for sub-agents does the same for their tools. On half of the MCP tasks graff now finishes in two model calls, where the release before needed three to five.

write_file is safer, and says what it did. It will not replace a file the session has not read unless the call says replace: true, so the model no longer checks whether a path is free before writing. Its result says whether it created or replaced the file and whether a .json file parses, so the model stops reading back what it just wrote.

Sub-agents can use their parent's MCP tools at once instead of starting with a tool search. The task that fans out to two sub-agents went from about 65 seconds to about 50.

Unattended runs skip what nobody reads. Under -p or in a piped session, graff no longer offers a question tool nobody can answer, and at low effort it skips reasoning summaries that only a screen would show.

Two ideas did not survive measurement: letting the final write and the "done" signal share one response (the model never did it), and telling the model that a batch of tool calls runs in order (it started splitting edits apart instead).

Sources: [adr-0231: ADR 0231: no round trip the model does not need](https://github.com/justrach/codegraff/blob/main/docs/adr/0231-no-round-trip-the-model-does-not-need.md); [release: graff v0.0.302.13](https://github.com/justrach/codegraff/releases/tag/v0.0.302.13)

## How we ran it

Every task runs in a fresh git repository with a fresh home directory, so no harness sees user-level settings or MCP servers. Harnesses run interleaved, task by task, a few at a time, each after one unscored warm-up request. Wall time is the runner's clock around each process, from start to exit.

List costs use gpt-6.1-sol's public rates: $2 per million input tokens, $0.10 per million cached and $10 per million output. A ChatGPT plan is not billed per token; the dollars make the harnesses comparable, not a bill.

Through the proxy, Claude Code's own prompt-cache markers do not survive, and only 49% of its MCP input was read from cache against graff's 96%. Against an endpoint that takes Anthropic's API directly, Claude Code's MCP times would likely be better, so treat its MCP gap here as an upper bound.

OpenCode 2 could not be signed into a ChatGPT plan without an interactive login, so it appears only in the gateway comparison. Three runs per task on one machine is a small sample: a second or two on any one task is noise, but the suite-level gaps held in every run that day.

Sources: [codex-results: graff vs the Codex app server and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/RESULTS.md); [cc-results: graff vs Claude Code, OpenCode 2 and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-claude-code/RESULTS.md); [graff-evals: graff-evals: tasks, harness drivers and runner](https://github.com/justrach/codegraff/tree/main/graff-evals)

## Since then

These numbers are from graff v0.0.302.13, on October 1. On gpt-6-astra, the build that became v0.0.302.14 was faster than the Codex app server on every suite we ran, including tasks that ask for sub-agents, at 16% to 73% lower cost. v0.0.302.15 made DeepSeek V4 Pro faster at graff's default effort.

Sources: [astra-results: graff vs the Codex app server on gpt-6-astra, with sub-agents](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/ASTRA.md); [deepseek-post: DeepSeek gets to work sooner in graff, and funded companies need a license](https://codegraff.com/blog/graff-0-0-302-16)

## Try it

graff update installs the latest release. The tasks, the harness drivers and every run's data are in the codegraff repo, so you can rerun both comparisons with graff-evals on your own tasks and models.

Sources: [release: graff v0.0.302.13](https://github.com/justrach/codegraff/releases/tag/v0.0.302.13); [graff-evals: graff-evals: tasks, harness drivers and runner](https://github.com/justrach/codegraff/tree/main/graff-evals)

## Sources

Every number comes from the public eval write-ups in the codegraff repo: 21 tasks, 3 runs of each per harness, harnesses interleaved task by task, each at its own defaults. The Codex comparison ran gpt-6.1-sol on one ChatGPT account. The gateway comparison ran one model through the Codegraff gateway, with the same model id and key for every harness; Claude Code reached it through a local proxy because the gateway takes chat completions only. List costs use gpt-6.1-sol's public rates; the gateway comparison reports tokens, not dollars.

- [codex-results: graff vs the Codex app server and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/RESULTS.md)

- [cc-results: graff vs Claude Code, OpenCode 2 and pi (results and every run)](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-claude-code/RESULTS.md)

- [adr-0231: ADR 0231: no round trip the model does not need](https://github.com/justrach/codegraff/blob/main/docs/adr/0231-no-round-trip-the-model-does-not-need.md)

- [release: graff v0.0.302.13](https://github.com/justrach/codegraff/releases/tag/v0.0.302.13)

- [graff-evals: graff-evals: tasks, harness drivers and runner](https://github.com/justrach/codegraff/tree/main/graff-evals)

- [astra-results: graff vs the Codex app server on gpt-6-astra, with sub-agents](https://github.com/justrach/codegraff/blob/main/evals/graff-vs-codex-app-server/ASTRA.md)

- [deepseek-post: DeepSeek gets to work sooner in graff, and funded companies need a license](https://codegraff.com/blog/graff-0-0-302-16)
