A coding agent is two things: the model, and the harness around it that decides what the model sees, which tools it can call and how many round trips a task takes. Hold the model fixed and whatever difference you measure is the harness. So we gave graff, OpenAI's Codex app server, Claude Code, OpenCode 2 and pi the same models and the same 21 tasks, three runs each. graff finished in 22% less time than the Codex app server and 34% less than Claude Code, almost all of it on MCP work. On small coding tasks, Claude Code and pi are still a little quicker.
The short version
We ran 21 tasks: 11 small coding jobs (fix a bug, rename a function, write tests, transform JSON) and 10 jobs that read a Linear-shaped MCP server, fetching issues and their comments and writing a report. Every run had to pass the task's own check, and every harness passed every run.
There are two comparisons, because no one model runs in all five harnesses: graff against the Codex app server on gpt-6.1-sol with one ChatGPT account, and graff against Claude Code, OpenCode 2 and pi on one model served through the Codegraff gateway.
- 22%less time per task than the Codex app server, same model and account
- 34%less time per task than Claude Code, same gateway model
- 2×faster than Claude Code on the MCP tasks
- 69%lower list cost per task than the Codex app server
How to read it. Each dot is a harness running the same model on the same 21 tasks; lower left is faster and cheaper. The coral line is the Pareto frontier: the best trade-offs on offer, where nothing beats a dot on both axes at once. Anything in the shaded area is beaten on both, slower and costlier than something on the line, so no trade-off makes it the better pick. On both models graff is the frontier, and every other harness lands in its shaded area.
gpt-6.1-sol, one ChatGPT account
One gateway model
Sources: graff vs the Codex app server and pi (results and every run) · graff vs Claude Code, OpenCode 2 and pi (results and every run)
Against the Codex app server
OpenAI's Codex app server is the engine inside the Codex apps. We drove it through its JSON-RPC interface on gpt-6.1-sol, with the same ChatGPT account graff used, both at the model's default effort.
graff took 22.2 seconds per task against 28.6, and the gap is about the same on both kinds of work: 16.0 against 20.4 seconds on the coding tasks, 29.0 against 37.8 on the MCP tasks. graff made 3.1 model calls per task to the app server's 4.4, and 3.3 to 5.8 on the MCP tasks, where one rlm script often fetches, trims and writes the report in a single step.
It cost less too: $0.013 per task against $0.043 at list price, 69% less. That is a floor for the app server, which also sends one warm-up request per thread that its usage does not report.
pi, a minimal open-source harness, has no MCP client, so it ran the coding tasks only: 17.0 seconds per task against graff's 16.0. Its prompt is about a tenth the size of graff's, so it costs less there.
See the numbersShow ▾Hide ▴
| Harness | All tasks | Coding | MCP | Calls per task | List cost per task |
|---|---|---|---|---|---|
| graff | 22.2s | 16.0s | 29.0s | 3.1 | $0.013 |
| Codex app server | 28.6s | 20.4s | 37.8s | 4.4 | $0.043 |
| pi | — | 17.0s | no MCP client | 3.2 (coding) | $0.007 (coding) |
Sources: graff vs the Codex app server and pi (results and every run)
Against Claude Code, OpenCode 2 and pi
Claude Code, OpenCode 2 and pi ran on one model served through the Codegraff gateway, with the same model id and key for every harness. Claude Code speaks only Anthropic's Messages API and the gateway takes OpenAI-style chat completions, so Claude Code reached it through a local LiteLLM proxy.
graff took 8.7 seconds per task, Claude Code 13.2 and OpenCode 2 14.1. On the MCP tasks graff took 10.6 seconds against 21.2 and 22.1: half the time.
On the coding tasks the order flips. Claude Code and pi took 6.0 seconds per task, OpenCode 2 6.8 and graff 7.0, and about 0.3 seconds of graff's is one slow gateway response on a one-call task. Small tasks on this model are where graff has the least to gain.
See the numbersShow ▾Hide ▴
| Harness | All tasks | Coding | MCP | Calls per task | Prompt tokens per MCP task |
|---|---|---|---|---|---|
| graff | 8.7s | 7.0s | 10.6s | 3.5 | 44k |
| Claude Code | 13.2s | 6.0s | 21.2s | 3.5 | 179k |
| OpenCode 2 | 14.1s | 6.8s | 22.1s | 4.3 | 63k |
| pi | — | 6.0s | no MCP client | 2.8 (coding) | — |
Sources: graff vs Claude Code, OpenCode 2 and pi (results and every run)
Where the time goes
The MCP gap is mostly context. Claude Code keeps every MCP result whole, so each call re-sends everything fetched so far: 179k prompt tokens per MCP task against graff's 44k, about 40k per call against 10k. It also wrote nearly twice the output, 1,645 tokens per MCP task against 882.
graff trims large list results before they reach the model, and often runs the fetch, filter and write loop as one rlm script, so the model reads less and is called fewer times. OpenCode 2 sits in between at 63k prompt tokens per MCP task, but made the most calls, 5.5 per task.
Task by task, graff was the fastest of the three on all 10 MCP tasks.
Sources: graff vs Claude Code, OpenCode 2 and pi (results and every run)
What changed in graff
Each of these removes a model call that did no work the task asked for, and every call costs seconds.
Servers you name load up front. When a prompt names an MCP server ("the linear server"), graff loads its tools with the first request instead of spending a round trip on a load call, and asking for sub-agents does the same for their tools. On half of the MCP tasks graff now finishes in two model calls, where the release before needed three to five.
write_file is safer, and says what it did. It will not replace a file the session has not read unless the call says replace: true, so the model no longer checks whether a path is free before writing. Its result says whether it created or replaced the file and whether a .json file parses, so the model stops reading back what it just wrote.
Sub-agents can use their parent's MCP tools at once instead of starting with a tool search. The task that fans out to two sub-agents went from about 65 seconds to about 50.
Unattended runs skip what nobody reads. Under -p or in a piped session, graff no longer offers a question tool nobody can answer, and at low effort it skips reasoning summaries that only a screen would show.
Two ideas did not survive measurement: letting the final write and the "done" signal share one response (the model never did it), and telling the model that a batch of tool calls runs in order (it started splitting edits apart instead).
Sources: ADR 0231: no round trip the model does not need · graff v0.0.302.13
How we ran it
Every task runs in a fresh git repository with a fresh home directory, so no harness sees user-level settings or MCP servers. Harnesses run interleaved, task by task, a few at a time, each after one unscored warm-up request. Wall time is the runner's clock around each process, from start to exit.
List costs use gpt-6.1-sol's public rates: $2 per million input tokens, $0.10 per million cached and $10 per million output. A ChatGPT plan is not billed per token; the dollars make the harnesses comparable, not a bill.
Through the proxy, Claude Code's own prompt-cache markers do not survive, and only 49% of its MCP input was read from cache against graff's 96%. Against an endpoint that takes Anthropic's API directly, Claude Code's MCP times would likely be better, so treat its MCP gap here as an upper bound.
OpenCode 2 could not be signed into a ChatGPT plan without an interactive login, so it appears only in the gateway comparison. Three runs per task on one machine is a small sample: a second or two on any one task is noise, but the suite-level gaps held in every run that day.
Sources: graff vs the Codex app server and pi (results and every run) · graff vs Claude Code, OpenCode 2 and pi (results and every run) · graff-evals: tasks, harness drivers and runner
Since then
These numbers are from graff v0.0.302.13, on October 1. On gpt-6-astra, the build that became v0.0.302.14 was faster than the Codex app server on every suite we ran, including tasks that ask for sub-agents, at 16% to 73% lower cost. v0.0.302.15 made DeepSeek V4 Pro faster at graff's default effort.
Sources: graff vs the Codex app server on gpt-6-astra, with sub-agents · DeepSeek gets to work sooner in graff, and funded companies need a license
Try it
graff update installs the latest release. The tasks, the harness drivers and every run's data are in the codegraff repo, so you can rerun both comparisons with graff-evals on your own tasks and models.
Sources: graff v0.0.302.13 · graff-evals: tasks, harness drivers and runner
Sources and method
Every number comes from the public eval write-ups in the codegraff repo: 21 tasks, 3 runs of each per harness, harnesses interleaved task by task, each at its own defaults. The Codex comparison ran gpt-6.1-sol on one ChatGPT account. The gateway comparison ran one model through the Codegraff gateway, with the same model id and key for every harness; Claude Code reached it through a local proxy because the gateway takes chat completions only. List costs use gpt-6.1-sol's public rates; the gateway comparison reports tokens, not dollars.
- graff vs the Codex app server and pi (results and every run)
- graff vs Claude Code, OpenCode 2 and pi (results and every run)
- ADR 0231: no round trip the model does not need
- graff v0.0.302.13
- graff-evals: tasks, harness drivers and runner
- graff vs the Codex app server on gpt-6-astra, with sub-agents
- DeepSeek gets to work sooner in graff, and funded companies need a license