# Graff evaluations

Every Graff coding-agent evaluation: same-model runs against Devin, Codex, Claude Code, OpenCode and Pi, release and model comparisons, FrontierHarness terminal tasks, DeepSWE patch verification, and a tuner for your own team.

Canonical: https://codegraff.com/evaluations

Original article published: 2026-09-06

Report updated: 2026-10-04. These are publication dates, not dates of a new evaluation run.

## Same model, other harnesses

### graff and Devin CLI on the same model (Oct 4, 2026)

gpt-6.1-sol, medium effort; 7 coding tasks × 3 runs, interleaved; seconds per task.

- graff v0.0.302.21: 46s
- Devin CLI: 128.2s

2.8× faster and 5.7× cheaper per task ($0.0249 against $0.1416 list). Devin passed one more run: 19/21 against 18/21. [Write-up](https://codegraff.com/blog/graff-vs-devin)

### Shorter turns against Pi's codemode (Oct 3, 2026)

gpt-6.1-sol; 11 tasks × 3 runs, every run passed; seconds per task.

- graff v0.0.302.18: 33s
- Pi 1.0 + codemode: 34.2s
- graff v0.0.302.17: 40.8s

32% lower list cost per task than Pi with codemode. On gpt-6-astra Pi stays 0.7 s faster on the mean; graff is 21% cheaper. [Write-up](https://codegraff.com/blog/graff-shorter-turns)

### Pi 1.0's codemode against graff's rlm (Oct 2, 2026)

gpt-6-astra; 11 tasks × 3 runs per setup; seconds per task.

- Pi + codemode: 41.7s
- Pi: 48.1s
- graff v0.0.302.17: 48.3s
- graff v0.0.302.16: 54.1s

Pi with codemode was the fastest of five setups here. The rlm change in v0.0.302.17 cut MCP join tasks by 27%, and v0.0.302.18 then overtook Pi. [Write-up](https://codegraff.com/blog/graff-rlm-vs-pi-codemode)

### graff against the Codex app server (Oct 1, 2026)

gpt-6.1-sol, same ChatGPT account; 21 tasks × 3 runs, every run passed; seconds per task.

- graff: 22.2s
- Codex app server: 28.6s

22% less time and 69% lower list cost per task ($0.0131 against $0.0426). [Write-up](https://codegraff.com/blog/graff-vs-codex-and-claude-code)

### graff against Claude Code and OpenCode 2 (Oct 1, 2026)

one gateway model; 21 tasks × 3 runs, every run passed; seconds per task.

- graff: 8.7s
- Claude Code: 13.2s
- OpenCode 2: 14.1s

Half the time on MCP work and 70% fewer prompt tokens than Claude Code; a little slower on the small coding tasks (7.0 s against 6.0 s). [Write-up](https://codegraff.com/blog/graff-vs-codex-and-claude-code)

## Releases and models

### DeepSeek gets to work sooner (Oct 2, 2026)

DeepSeek V4 Pro; 21 coding and MCP tasks × 2 runs; seconds per task.

- v0.0.302.15: 15.9s
- v0.0.302.14: 21.6s

26% less time per task, 39% less on sub-agent tasks, 36% fewer output tokens. [Write-up](https://codegraff.com/blog/graff-0-0-302-16)

### Grok 4.6 against Grok 4.7 in graff (Sep 22, 2026)

Grok 4.6 / 4.7; 3 real pull requests, 24-point checklist; wall clock for all three PRs.

- Grok 4.6: 507s
- Grok 4.7: 1351s

4.7 scored one checklist point higher (21/24) for 2.3× the cost and 3.4× the reasoning tokens. [Write-up](https://codegraff.com/blog/grok-4-6-vs-4-7)

### The rlm runtime, before and after (Aug 28, 2026)

one task, reused binds; same task, rlm off then on; seconds for the task.

- With rlm: 39.2s
- Before: 99.6s

Input tokens 135k → 25.3k and model calls 11 → 5 on the same task. [Write-up](https://codegraff.com/blog/graff-rlm-token-efficiency)

Results are repository-reported evaluations checked against the tracked summaries and protocol at revision b4bd80c, not a fresh rerun or an independent audit. The Graff terminal runs shown here used additional evaluation instructions. Models and execution environments differ across some rows. Published Exo costs are taken from FrontierHarness revision 8f11b13; the terminal subtotal is calculated from its public task-level records.

## Interpreting the results

The 21 selected Terminal-Bench 2.1 tasks and nine DeepSWE patch tasks test different outcomes. Both Graff DeepSWE runs passed KaTeX multicolumn-array-spans; the other eight tasks did not pass. Missing verifier results count as failures.

Graff received additional terminal instructions and used a local Docker runtime. The published board used Kimi K3 with shared runtime and instructions. These configurations do not isolate a harness-only effect. Costs include failed attempts; Graff uses historical list-price estimates while the published board uses first-turn-cold pricing. Partial costs are lower bounds, not zero. Graff median attempt times are not reported.

### Find your tradeoff.

Twelve published Kimi K3 configurations plus two recorded Graff runs. Graff used extra terminal instructions and a local Docker runtime; its full-suite totals combine the two evaluated slices. Published costs use cost_first_cold_usd; Graff costs use historical list-price summaries. All recorded failures count toward spend. Partial published costs are lower bounds. Frontier badges consider only fully costed published configurations with at least one pass. Median attempt time includes successes and failures where task timings are available; Graff summary maxima are not medians.

#### All 30 tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 20/30 | $69.3659 | $3.47 | 30/30 | 9m 22s | Yes |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 19/30 | ≥ $62.4132 | ≥ $3.28 | 27/30 | 12m 29s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 19/30 | ≥ $348.3998 | ≥ $18.34 | 29/30 | 18m 29s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 18/30 | $43.7934 | $2.43 | 30/30 | 10m 39s | Yes |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 18/30 | ≥ $82.3941 | ≥ $4.58 | 27/30 | 8m 54s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 18/30 | ≥ $62.2600 | ≥ $3.46 | 27/30 | 9m 53s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 17/30 | $80.6742 | $4.75 | 30/30 | 9m 56s | No |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 17/30 | $62.0005 | $3.65 | 30/30 | 9m 20s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 17/30 | ≥ $80.2052 | ≥ $4.72 | 26/30 | 14m 54s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 16/30 | $16.7227 | $1.05 | 30/30 | 10m 04s | Yes |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 15/30 | ≥ $48.6652 | ≥ $3.24 | 29/30 | 11m 41s | No |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 15/30 | ≥ $43.5646 | ≥ $2.90 | 28/30 | 13m 06s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 21/30 | $31.7951 | $1.51 | 30/30 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 18/30 | $25.5611 | $1.42 | 30/30 | Not reported | No |

#### 21 terminal tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 15/21 | $4.8584 | $0.32 | 21/21 | 6m 51s | No |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 15/21 | ≥ $3.3749 | ≥ $0.22 | 18/21 | 7m 33s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $15.5788 | $0.97 | 21/21 | 9m 38s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $5.4599 | $0.34 | 21/21 | 8m 06s | No |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 16/21 | ≥ $3.0198 | ≥ $0.19 | 18/21 | 7m 55s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 16/21 | ≥ $8.1457 | ≥ $0.51 | 19/21 | 7m 04s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $5.4428 | $0.34 | 21/21 | 7m 24s | Yes |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 14/21 | $9.3918 | $0.67 | 21/21 | 7m 56s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 15/21 | ≥ $6.2995 | ≥ $0.42 | 17/21 | 6m 55s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $6.4250 | $0.40 | 21/21 | 6m 51s | No |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 15/21 | $3.4771 | $0.23 | 21/21 | 7m 43s | Yes |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 13/21 | $11.3272 | $0.87 | 21/21 | 9m 08s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 20/21 | $6.2335 | $0.31 | 21/21 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 17/21 | $3.6463 | $0.21 | 21/21 | Not reported | No |

#### 9 DeepSWE tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 5/9 | $64.5075 | $12.90 | 9/9 | 43m 07s | Yes |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 4/9 | $59.0383 | $14.76 | 9/9 | 37m 12s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 3/9 | ≥ $332.8210 | ≥ $110.94 | 8/9 | 59m 56s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $38.3334 | $19.17 | 9/9 | 39m 11s | No |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $79.3743 | $39.69 | 9/9 | 50m 21s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 2/9 | ≥ $54.1143 | ≥ $27.06 | 8/9 | 47m 16s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 1/9 | $75.2314 | $75.23 | 9/9 | 49m 57s | No |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 3/9 | $52.6087 | $17.54 | 9/9 | 44m 29s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $73.9058 | $36.95 | 9/9 | 41m 25s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 0/9 | $10.2977 | No passes | 9/9 | 15m 55s | No |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 0/9 | ≥ $45.1881 | No passes | 8/9 | 49m 05s | No |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 2/9 | ≥ $32.2374 | ≥ $16.12 | 7/9 | 28m 48s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 1/9 | $25.5616 | $25.56 | 9/9 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 1/9 | $21.9148 | $21.91 | 9/9 | Not reported | No |

## Read the experiment

[Full evaluation article and methodology](https://codegraff.com/blog/graff-frontier-harness-evals)

## Sources

- [FrontierHarness results and suite definitions](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/README.md)
- [Evaluation protocol and comparison limitations](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/PROTOCOL.md)
- [Graff / Grok terminal result summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/results.maxima.json)
- [Graff / Kimi K3 terminal result summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/kimi-results.maxima.json)
- [Published FrontierHarness task costs and Exo results](https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/results/eval-data.json)
- [Published cost accounting and comparison method](https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/skills/frontierharness-eval/reference.md#cost-comparability)
- [Graff / Grok DeepSWE cost summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/swe-results.maxima.json)
- [Graff / Kimi K3 DeepSWE cost summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/swe-kimi-results.maxima.json)