Four code-navigation products. Two kinds of readiness.

The most important methodological decision

“Ready” does not mean one thing

The primary metric is therefore time to an asserted navigation result with the completion boundary attached. Empty product caches were used; the operating system page cache was uncontrolled and potentially warm.

Atlas Scout · CodeGraphContext

When the finish line is a completed persistent index

On both corpora where both products completed, Atlas Scout reached the declared finish line first. On Kodi the gap is felt rather than read: 11.6 seconds against eleven and a half minutes. The large-corpus outcome is equally important: Scout finished all Firefox and Linux repetitions; default CodeGraphContext reached the frozen Linux usefulness ceiling before a qualifying result.

CodeGraphContext documents KuzuDB as a larger-capacity embedded alternative. It was not silently substituted for the frozen default FalkorDB Lite configuration; a KuzuDB run belongs in a separately labeled tuned campaign.

Serena · mcpls · prepared engine state

A known-file answer can arrive before the workspace is fully understood

The LSP-backed products demonstrate a different—and genuinely compelling—experience. With the required project metadata prepared, mcpls returned the asserted Firefox symbol in under a second. Serena reached the same-file milestone in under a minute on every corpus.

Stronger Serena cross-file milestones

Equivalent strong cross-file pilots were not run for mcpls, so its values are not measured—not inferred from Serena and not treated as zero.

Setup is not query latency—but it is still user time

The clean checkout changes the story

Serena’s official C/C++ guidance requires a correctly configured compile_commands.json. Because mcpls also launches clangd and recognizes that metadata, both products received the same hash-locked database and isolated clangd cache. Atlas Scout and CodeGraphContext required no corpus build preparation.

Speed, memory, CPU, writes, and disk

The fast complete map is not the cheap map

Scout’s large-corpus completion came with substantial resource use. CodeGraphContext’s completed indexes were much smaller and its builds stayed near one logical core. The LSP products used little persistent cache by comparison, but their process trees continued working after the first result and reached multi-gigabyte memory peaks.

Cold LSP phase resources

LSP memory can peak after the readiness milestone. Average CPU covers the complete measured phase, including a frozen 30-second live observation—not only the named-file query.

AT001 · 45 valid runs · five arms · three agent hosts

Correctness first. Then time, tokens, and whether the tool contributed.

A fast index is worth nothing if the agent holding it still answers wrong. A separate campaign asked each agent to locate Ollama’s (*Server).ChatHandler, report its exact inclusive range, and explain the opening request-validation behavior from source. Every host ran shell-only navigation and all four MCP products three times in a frozen order.

Overall, 37 of 45 answers earned 5/5. Every answer earned full behavior and source evidence credit; the remaining eight missed only the exact-range points, and all eight came from Claude Code.

Arm-level outcome

Host-native token accounting

These are whole-agent totals: instructions, tool catalogs, cached context, navigation, source reads, retries, reasoning, and final answers. Each host uses a different model, tokenizer, cache scheme, and usage schema, so the only honest comparisons are within one host.

Did the assigned navigation product actually contribute?

A correct final answer does not prove the assigned MCP product helped. Agents can fall back to shell and source reads after an empty result, cancellation, or startup failure. The routing record keeps product contribution separate from final-answer correctness.

Serena also returned non-empty output on every call. CodeGraphContext contributed non-empty results in all three Claude runs, while its other host paths were empty or cancelled. mcpls contributed four non-empty Claude results; its Antigravity discovery incompatibility produced three valid startup-failure outcomes, and its Codex calls were cancelled. Those are host-product interoperability results—not discarded harness failures.

Authoritative revision-4 lock SHA-256: 34e4696febd67c9f3df9bcf91be4cca53d1ea14b6b74e7ef7e39cad2c240cdceToken-augmented summary SHA-256: 8318ddadb2a22bc30ffb016ac600cf3c146734a0767de15cb202fd70d4d61d06

This extension does not rewrite the original 48-position resource campaign or 45-run agent study. It separately tested CodeGraphContext’s KuzuDB backend, added Claude Opus 5, and added three OpenCode model lanes through the operator-owned ZaguanAI gateway with Fireworks as their common downstream inference provider.

Kuzu was stopped for host safety before readiness

CodeGraphContext 0.5.2 with KuzuDB 0.11.3 did not finish indexing Ollama, the smallest frozen corpus. At 44 minutes 35.810 seconds, the resource guard fired as host-available memory crossed the frozen 12 GiB floor. This was not a query-quality failure: no ready index existed to query.

Blinded correctness across the four new host/model lanes

All 60 accepted runs answered the task, preserved the source worktree, and passed model, transcript, mutation, credential, and process-cleanup gates. Every answer correctly explained the request-validation behavior. The four non-perfect answers all contained exact-range errors; one also shifted the cited validation lines.

Product contribution remained separate from answer correctness

Every observed MCP call in the accepted extension completed with non-empty output. Scout used one MCP call per run; the other products used between 17 and 32 calls across their 12 runs.

Token leaders—inside each host/model only

Hosts and models use different tokenizers, prompts, caches, and accounting schemas. These values therefore identify the lowest median inside each frozen lane; they are not pooled into a cross-model token leaderboard.

Claude Opus shell’s uncached median was only 29 tokens below Scout’s 6,439. OpenCode reported zero cost because the custom provider configuration contained no pricing metadata; that is not evidence that ZaguanAI, Fireworks, or the model invocation was free.

Accepted extension provenance

Kuzu resource lock SHA-256

cbccbc3a6f125626d1e73afa8d4ba81a6640b789eafb3bc7d520864575bdfbf7

Kuzu aggregation SHA-256

a1c4e030cae1d9742e63130efd8420ac89863d89108f787acc751f3e2f7ad10f

Accepted extension order SHA-256

3e63989d14a37414600dd0b792751f4369fc4ec9159aaa0e3a6f7918ce63415b

Accepted summary SHA-256

bed1c017543d4942e022f385b2d4b89ddaa186018d16a1f8405f3eb1bb12d0e3

Accepted runner SHA-256

5c4f7990da4f34385def3dcb9f3cb2c4010a196ca06a48e03c500fcdbc858d03

The formal report, frozen locks, invalidated revision, all 60 accepted reruns, deterministic analyses, and retained raw evidence are published in the benchmark evidence repository.

No grand score · no loser column

What each project demonstrated

Frozen protocol · deterministic aggregation · no outlier removal

How the campaign was kept honest

Frozen corpora

Formal host

CPU

AMD Ryzen 9 5950X · 16 cores · 32 threads

Memory

31.23 GiB OS-visible RAM

Storage

Samsung 980 PRO 1-TB NVMe · ext4 · BFQ

System

Mageia 11 · Linux 6.18.35 · glibc 2.42

Cache condition

Empty product cache; uncontrolled, potentially warm OS page cache

Order

Frozen and counterbalanced across three attempts

Every product command and its descendants ran in an isolated cgroup. The harness measured complete process trees, retained every observation, and sampled host memory and swap. The campaign began after reboot with zero swap; the observed maximum stayed below 50 MiB and the frozen 256-MiB launch gate.

Campaign identity

Campaign

expanded-resource-001-preview21-r5

Frozen seed

20260729

Campaign lock SHA-256

cdd8598516f6b1c2f8f3929d0553cf81979133ab39ae301799bb00f7717f9004

Aggregation SHA-256

f8218f338ab938292a9c7a1bdc85785e4894c0ec9f047a550048a17d52b42d95

Campaign runner SHA-256

92e396cfa00199d16f457e4bf7b2bff4dd78b88725b2e8169dfd846fa65e88bd

Campaign analyzer SHA-256

b8dc8732f536ee27e3f486a11e8e64d2e30b603675de75ced6a5fc560b0ea510

The 48 immutable lane manifests, machine-readable aggregation, included raw runs, and normalized analyses are published in the public evidence repository. The repository also records the precise boundary between published evidence and excluded reproducible or third-party artifacts.

What this page cannot honestly claim

A detailed result is still a bounded result

• This is internal formal evidence produced by Atlas Scout’s developer, with maintainer factual review still open.
• Each completed cell has three repetitions on one workstation and one frozen commit per corpus. The OS page cache was not controlled.
• The primary anchor is a known-file same-file query. This campaign does not by itself rank relationship correctness, fuzzy discovery, incremental updates, warm latency, or editing quality.
• Product installation time, download bytes, and complete dependency footprints were not measured comparably and are excluded from the clean-checkout stage sums.
• mcpls strong cross-file pilots were not run. Sourcegraph was not part of the local category.
• The additive agent extension remains one exact-symbol task on one Ollama commit. Its four host/model lanes and three repetitions per cell do not establish a general model or product ranking.
• Kuzu never produced a ready index, so its 21 conditional agent cells are unavailable. The campaign does not score Kuzu query correctness, latency, or token use.
• OpenCode’s zero reported cost reflects missing pricing metadata, not free inference. Token units are compared only inside one frozen host/model lane.
• No single “best code tool,” “fastest indexer,” or grand score is supported by this evidence.

What should happen next

1. Send each maintainer their configuration and product-specific draft results.
2. Correct verified factual mistakes and publish substantive maintainer responses.
3. Invite CodeGraphContext maintainers to review the frozen KuzuDB configuration and safety result before drawing broader backend conclusions.
4. Run a frozen mcpls strong cross-file campaign.
5. Capture comparable product-install time, downloads, and dependency footprint.
6. Reduce Scout’s large-corpus index size, peak memory, and write amplification without sacrificing complete-readiness time.

The benchmark did what it was supposed to do: it found a real Atlas Scout advantage, several real competitor advantages, and the next engineering work Scout has not earned the right to avoid.

Public evidence repository

Inspect the campaign, not only this interpretation