Most code-search benchmarks let a model judge itself. We tried the opposite: a deterministic oracle that asks one yes/no question — did the agent name the file the merged PR actually patched?
TL;DR
- 4 SWE-bench Lite tasks × 6 backends, scored by file-path match. No LLM judge.
- codedb_CONTEXT sits at the efficient end of the matched-quality cluster (3/4 top-1, 4/4 recall).
- Three releases on the way: v0.2.5815 found two regex outliers, v0.2.5816 fixed them (20× p50 / 10× p99), v0.2.5817 shipped an auto-prepended codebase map.
- Hypothesis snapshot, not a dominance claim. n=4 is a sanity check. Data + scorer in PR #495.
How we got here
Before we look at the bench: the same warm tree query that took 5.5 seconds on v0.2.53 is under 100 ms now — a roughly 60× swing over seven weeks. That's the runway the bench is sitting on.
The bench
Four SWE-bench Lite instances, each pinned at the base_commit the upstream issue was filed against:
| Instance | Gold file (the merged PR's patch site) |
|---|---|
pallets__flask-4045 | src/flask/blueprints.py |
psf__requests-2148 | requests/models.py |
psf__requests-2674 | requests/adapters.py |
mwaskom__seaborn-2848 | seaborn/_oldcore.py |
Scoring — deterministic, no model in the loop:
- recall — gold file appears anywhere in the agent's
fileslist - top-1— agent's first listed file equals the gold file
The six backends — what each one is:
| Backend | What it is |
|---|---|
codedb | codedb CLI primitive — agent issues raw search / ls calls |
codedb_CONTEXT | codedb's MCP composer — one call returns a ranked file slice |
leanctx | peer context tool, agent loop over its primitives |
fts5_trigram | SQLite full-text baseline — naive but exhaustive |
codegraph | CLI primitive from colbymchenry/codegraph |
codegraph_CONTEXT | codegraph's composer |
Headline numbers
Rows are sorted by top-1. Quality on the left, cost on the right. The wall column is what to watch.
Quality dots: each circle = 1 task out of 4 (filled = correct). * codegraph rows reflect a fixed-query subprocess measurement, not an LLM agent loop — their wall and tokens are not directly comparable to the other rows.
What jumps out
Quality is mostly uniform.
Five of six backends recall the gold file on all four tasks. Top-1 splits across one ambiguous task: fts5_trigram 4/4, four others tied at 3/4.
codegraph_CONTEXT is the lone quality outlier.
Misses both `requests` tasks. The issue text mentions urllib3 keywords; the composer surfaces urllib3 internals over the requests-layer wrapper where the patch actually lands. Reproducible across codegraph v0.7.10 → v0.9.3.
codedb_CONTEXT is the efficient end of the matched-quality cluster.
Matches codedb / leanctx at 3/4 top-1 and is the cheapest in the cluster across calls, wall, and tokens. fts5_trigram is the only 4/4 top-1 — at ~20× the wall.
codegraph_CONTEXT requests-task miss
Issue text mentions
socket.errorurllib3DecodeError
Surfaces (urllib3 layer)
- urllib3/connection.py
- urllib3/util/ssl_.py
- urllib3/exceptions.py
Gold patch (requests wrapper layer)
- requests/models.py ← psf__requests-2148
- requests/adapters.py ← psf__requests-2674
The composer ranks the actual urllib3 implementation higher than the requests-layer wrapper because that's where the symbols + calls anchor. The gold patches live one layer up. Reproducible across codegraph v0.7.10 → v0.9.3.
The seaborn split
Two valid orderings, one strict scorer:
- The error surfaces in
seaborn/_oldcore.py(where the merged patch lands). - But the call site a developer would trace lives in
seaborn/axisgrid.py.
Four backends named axisgrid.py first. Two named _oldcore.py first. Both orderings find the bug — top-1 just punishes the explanatory one.
Measurement caveat
Codegraph rows were measured differently:
- Calls / wall: subprocess invocations driven by a fixed query plan, not an LLM agent loop.
- Tokens: stdout bytes / 4, not agent context tokens.
So codegraph's quality cells are comparable; its efficiency cells are not.
What shipped on the way — v0.2.5815 → 5817
The shape was bench → fix → bench → confirm:
v0.2.5815
Cross-corpus head-to-head
First systematic run pitting codedb against codegraph, lean-ctx, and SQLite FTS5 baselines across react / regex / flask. Won 13/15 react warm queries vs codegraph — and surfaced two latency outliers on the regex suite that needed fixing.
found 2 regex outliers ≈ 10–20× slower than they should be
v0.2.5816
Tier 5 short-circuit + codedb read CLI
The regex fall-through ran a full content scan even when the trigram filter had already produced sound candidates. Removing that double-scan unlocked the outliers. Ships alongside the new `codedb read <path>` CLI with -L line ranges, --compact mode, project-root anchoring, and sensitive-file guards.
Suspense regex 2.82 ms → 0.14 ms (20× p50)
v0.2.5817
reader.md auto-prepended codebase map
codedb_context now auto-prepends a hash-verified reader.md codebase map. Gives the agent a single-pass orientation document (directory layout, key entry points, public API surfaces) before the task-specific context. Hash check ensures the map reflects the current tree — stale maps fail closed.
−57% calls, fewer fanout fetches, better first-shot file naming
The regex outliers, before and after
Two regex queries on v0.2.5815 were spending ~10 ms re-scanning content the trigram filter had already narrowed. v0.2.5816 short-circuits that path. 20× faster at p50, 10× at p99, recall unchanged.
Source: v0.2.5816 release notes, benchmarks/search-shootout, 20 warm iters per query. Recall preserved on every query; hit counts identical.
Hypothesis
Stated as something to falsify, not declare:
codedb_CONTEXT is the cheapest backend in the matched-quality cluster (3/4 top-1, 4/4 recall). fts5_trigram is the only 4/4 top-1, at ~20× the wall.
Top-1 = 4/4
best file-localization qualityTop-1 = 3/4
matched-quality clusterAvg wall time per task across 4 SWE-bench Lite instances. Codegraph rows omitted — they were measured under a fixed-query subprocess plan, not an LLM agent loop, so their wall is not comparable.
Falsifiable by:
- A live LLM-loop runner that re-measures codegraph at agent-loop methodology
- Expanding to 20–50 SWE-bench Lite instances so the differences become statistical
- A patch-correctness oracle: apply the agent's patch against the pinned
base_commitand run the failing tests
Until then, treat the headline as directional, not quantitative.
Reproduce
git clone https://github.com/justrach/codedb cd codedb && git checkout bench/publish-swe-lite-results python3 benchmarks/swe-lite/replay.py
The script exits non-zero if any summary cell drifts from the raw cells, so the numbers self-check. Files in benchmarks/swe-lite/:
results.json— 24 cells (6 backends × 4 tasks)replay.py— recomputes + verifies + prints the matrixRESULTS.md— the publishable read (this post is the friendlier version)