{"schemaVersion":1,"canonical":"https://codegraff.com/blog/graff-vs-opencode","markdown":"https://codegraff.com/blog/graff-vs-opencode/markdown","slug":"graff-vs-opencode","title":"Graff vs OpenCode: two more terminal passes, and the fine print","seoTitle":"Graff vs OpenCode: Coding Agent Benchmarks & Costs","description":"Compare Graff vs OpenCode on FrontierHarness: Kimi K3 task completion, cost per pass, DeepSWE results, and the limits of this coding agent benchmark.","publishedAt":"2026-09-19","keywords":["Graff vs OpenCode","FrontierHarness","coding agent benchmarks","Kimi K3","agent evaluation"],"intro":"I build Graff, so this Graff vs OpenCode comparison is one I want to get right. These FrontierHarness coding agent benchmarks compare completed tasks and recorded costs. With Kimi K3, our recorded run completed 17 of 21 terminal tasks; OpenCode’s published run completed 15. OpenCode spent slightly less in total. Graff’s recorded cost per pass was slightly lower. That is an interesting head-to-head, provided we keep the run conditions attached to the scores.","hero":{"src":"/blog/graff-frontier-harness-evals/eval-workshop.webp","alt":"Three mouse engineers test completed pieces from separate brass workbenches using one central checking jig.","caption":"The same checking jig is only part of a fair comparison. Models, instructions, and execution environments matter too. Conceptual illustration; scores appear in the charts and tables below."},"sourceNote":"This article revisits the historical results used in our September 6 FrontierHarness report. It does not report a new OpenCode run or benchmark the latest release. Graff summaries and public board data are pinned to commits in the sources below. Costs are historical recorded estimates, not current prices. No repeated-run uncertainty estimates are available in this comparison.","sections":[{"id":"the-matchup","title":"My harness, your harness, one useful question","paragraphs":["If you use OpenCode, the useful question is whether Graff completes work that your current setup leaves unfinished, at a cost you can justify. A brand comparison alone cannot answer that. We need tasks with an observable finish line.","FrontierHarness gives this comparison 30 tasks: 21 terminal tasks and nine DeepSWE patch-verification tasks. I separate them because completing a terminal workflow and producing a patch that passes a repository’s verifier are different tests.","OpenCode’s documentation describes an open-source coding agent available in a terminal, desktop app, and IDE extension. Those interaction choices matter in everyday use. This article measures task outcomes in the recorded evaluation; it does not score editor experience or claim to test OpenCode’s current release."],"sources":["results","opencode-docs"]},{"id":"scorecard","title":"Graff vs OpenCode benchmark results: 17 vs 15 terminal passes","paragraphs":["The closest model match in our stored results is Kimi K3 against Kimi K3. Graff completed 17 terminal tasks and OpenCode completed 15. On this 21-task slice, that is 81.0% against 71.4%, a difference of two tasks or 9.5 percentage points.","The model match is useful, but the setup still differs. Graff’s recorded run used additional terminal evaluation instructions and a local Docker runtime. OpenCode’s row comes from the published board. This comparison cannot isolate how much of the difference came from the harness, the instructions, or the execution environment.","Our Grok 4.6 run completed 20 of 21 terminal tasks. I include it for context, but switching the model changes the experiment. It would be misleading to turn that row into a claim that Graff’s harness alone beats OpenCode by five tasks."],"sources":["kimi-summary","grok-summary","published-board","protocol"],"table":{"caption":"Historical FrontierHarness results. Matching the model does not match the full protocol.","headers":["Configuration","Terminal tasks","DeepSWE tasks","Run conditions"],"rows":[["Graff / Kimi K3","17 / 21 (81.0%)","1 / 9","Extra terminal instructions; local Docker"],["OpenCode / Kimi K3","15 / 21 (71.4%)","0 / 9","Published board configuration"],["Graff / Grok 4.6","20 / 21 (95.2%)","1 / 9","Different model; extra terminal instructions; local Docker"]]}},{"id":"cost","title":"Cost comparison: total spend and cost per completed task","paragraphs":["Across the terminal tasks, OpenCode’s recorded total was $3.4771 and Graff/Kimi’s was $3.6463. OpenCode therefore spent about 17 cents less across the slice. Dividing each total by completed tasks gives about $0.2318 per pass for OpenCode and $0.2145 for Graff.","Cost per pass includes spending on failures. It answers a more useful question than the price of a successful attempt alone: how much did the whole evaluation spend for each completion it produced?","The numerical difference in cost per pass is about 7.5% in Graff’s favor, but it is not a controlled savings estimate. Graff used historical evaluator list prices; the public board uses its own first-turn cache repricing method. These are reported amounts under different accounting rules, not a promise about your bill.","This is also why I would not place the Graff row on the public board’s Pareto frontier. A frontier describes which configurations offer an unbeaten cost-and-completion tradeoff within comparable conditions. An attractive point from a different protocol does not establish that result."],"sources":["kimi-summary","published-board","board-cost-method","protocol"],"table":{"caption":"Terminal slice only: all 21 tasks have recorded costs in both rows. USD; rounded for display.","headers":["Kimi K3 configuration","Recorded total","Passes","Recorded cost / pass"],"rows":[["Graff","$3.6463","17","$0.2145"],["OpenCode","$3.4771","15","$0.2318"]]}},{"id":"patches","title":"DeepSWE results: verified repository patches","paragraphs":["Graff/Kimi passed one of the nine DeepSWE tasks. OpenCode/Kimi passed none. Graff’s successful task was KaTeX; the other eight failed. One verified patch is a real outcome, but eight misses leave substantial room to improve.","Adding the two slices produces 18 of 30 for Graff/Kimi and 15 of 30 for OpenCode/Kimi. Those totals are arithmetically correct, yet the terminal-only headline would conceal how weak the patch results were. For someone choosing an agent for repository bug fixes, the nine-task breakdown deserves its own line.","Full-suite cost needs another qualification: OpenCode has published costs for 29 of the 30 tasks, with one DeepSWE task missing a cost. Treating that missing value as zero would understate spending. The terminal comparison above avoids that missing-cost problem, though the accounting differences remain."],"sources":["results","kimi-swe-summary","published-board"]},{"id":"decision","title":"Is Graff an OpenCode alternative for your workflow?","paragraphs":["I would treat Graff’s terminal result as a reason to try it on a small set of representative tasks. I would not switch a working setup on the strength of two extra completions in one historical evaluation. These scores do not measure review effort, maintenance burden, or the usefulness of a tool in your daily workflow.","The next comparison should hold the model and provider, instructions, repository state, tool access, timeout, and cost accounting constant. Both agents should start from clean environments and be graded with the same verifier. Repeated attempts would show whether the gap survives run-to-run variation.","For my side of that experiment, the target is clear: preserve terminal completion while getting more repository patches through verification. For your side, bring the tasks that matter to your work. That would make the next Graff vs OpenCode article much more useful than another headline score."],"sources":["protocol","results"]},{"id":"which-is-better","title":"Is Graff better than OpenCode?","paragraphs":["In these historical Kimi K3 runs, Graff completed more terminal tasks: 17 of 21 compared with OpenCode’s 15. That does not establish a general winner. Instructions and runtime differed, and the comparison does not measure daily usability or the latest releases. Use the results to choose what to evaluate on your own tasks."],"sources":["kimi-summary","published-board","protocol"]},{"id":"which-costs-less","title":"Is Graff cheaper than OpenCode?","paragraphs":["OpenCode had the lower recorded terminal total: $3.4771 against Graff’s $3.6463. Graff had the lower recorded cost per completed task: $0.2145 against $0.2318. Different accounting methods prevent a controlled pricing claim; these figures are benchmark costs, not subscription prices or current provider quotes."],"sources":["kimi-summary","published-board","board-cost-method"]},{"id":"latest-version","title":"Does this benchmark test the latest OpenCode version?","paragraphs":["No. This article reuses the pinned FrontierHarness data from our earlier evaluation report. It is a historical comparison, not a new run of the latest OpenCode or Graff release. The source links identify the data snapshots behind the numbers."],"sources":["published-board","results"]}],"sources":[{"id":"results","title":"FrontierHarness results and suite definitions","url":"https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/README.md"},{"id":"protocol","title":"Evaluation protocol and comparison limitations","url":"https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/PROTOCOL.md"},{"id":"grok-summary","title":"Graff / Grok terminal result summary","url":"https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/results.maxima.json"},{"id":"kimi-summary","title":"Graff / Kimi K3 terminal result summary","url":"https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/kimi-results.maxima.json"},{"id":"published-board","title":"Published FrontierHarness task costs and Exo results","url":"https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/results/eval-data.json"},{"id":"board-cost-method","title":"Published cost accounting and comparison method","url":"https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/skills/frontierharness-eval/reference.md#cost-comparability"},{"id":"kimi-swe-summary","title":"Graff / Kimi K3 DeepSWE cost summary","url":"https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/swe-kimi-results.maxima.json"},{"id":"opencode-docs","title":"OpenCode official documentation (checked September 19, 2026)","url":"https://opencode.ai/docs/"}]}