Benchmarks

Two repositories, two lessons

Firefox and Linux stress different parts of Atlas Scout. Firefox exposed a resolver that did far too much work before the index became ready. Linux exposed the cost—and the payoff—of building a reusable search path for the questions an agent asks after indexing.

These are separate same-host measurements with different protocols. The Firefox result is the stronger end-to-end scaling test; Linux is the stronger controlled search tradeoff comparison.

Firefox · reboot-cold Preview 7 candidate

From two and a half hours to four minutes

We do not merely benchmark Atlas Scout to prove it is fast. We benchmark it to discover where it is wrong.

Preview 6 successfully indexed the Firefox checkout, but complete fresh readiness took 2 hours, 32 minutes, and 46 seconds. Nearly all of that time was spent resolving relationships across files.

The Preview 7 source candidate indexed the same checkout in 4 minutes and 0.43 seconds after a host reboot with a verified-empty cache: 38.1× faster end to end. The resolution phase fell from more than two and a half hours to 15.823 seconds, a 569.7× improvement.

A Firefox-sized structural map

The checkout contained 469,704 discovered files at Firefox revision f4e6e71e9c4deb3818880f2c8a22de09a92955ce. Atlas Scout indexed every supported file with zero failed files.

Interactive after the index is ready

The corrected search benchmark measured 500 sequential requests in one persistent MCP session: 25 distinct queries across five classes, repeated in four interleaved passes. Initialization, readiness checks, and one warm-up were excluded. All requests completed without tool errors.

A separate 300-request run measured a populated reference lookup at 2.92ms p50 and a 20-item file outline at 3.41ms p50. These timings include serializing useful model context, not only database lookup time.

Faster without giving the model less

The optimization changes how Scout narrows possible imported targets, not the rule for accepting a relationship. Preview 6 and the Preview 7 candidate produced the same 1,204 import-binding and 2,378 import-path resolutions. Ambiguous imports remain unresolved rather than guessed.

The ordered projection of relationship identity, resolved target, strategy, and confidence reproduced the expected SHA-256:

e2282fd551592a3d1e4b89e8940ef21b2f1b615e5bc0eb432572cb3cad90b354

Scoped searches also passed deliberately difficult probes: a common function named test was returned from the requested file, while a scoped search for Distance still found compound CamelCase names. The final database passed its integrity check.

The costs are part of the result

Preview 7 deliberately spends more during indexing so repeated agent queries do less:

• The generated database is 5.15% larger.
• Peak memory during the reboot-cold run is 4.0% higher.
• Extraction and commit take 49.1% longer before the much shorter resolution phase.
• Starting a fresh MCP process still takes about 5.7 seconds on this machine.
• A complete workspace schema summary remains roughly a 346ms operation.
• A same-profile no-change refresh took 8.32 seconds of process wall time, compared with 7.07 seconds in Preview 6.

The accepted trade is modestly more local storage and indexing work in exchange for removing the multi-hour readiness tail and making repeated navigation interactive.

Method and interpretation boundaries

Both completed databases used the same Firefox checkout and the same Linux workstation: an AMD Ryzen 9 5950X with 32 GiB of memory and a Samsung 980 PRO NVMe SSD. The Preview 7 command began roughly three minutes after reboot. No recursive operation had traversed the source tree, and the external Atlas Scout cache was new and verified empty.

The Preview 6 control used a fresh database on the same host and corpus, but it did not begin after reboot. Preview 6 used the public release binary; Preview 7 used a recorded local source candidate built with Rust 1.95.0, while release CI uses Rust 1.94.0. The measurements therefore validate the source candidate, not the byte identity or a universal performance guarantee for the eventual public artifact.

Query percentiles are nearest-rank observations from warm persistent MCP sessions. MCP initialization, readiness, and one warm-up were excluded from those distributions and reported separately.

The benchmark was tested too

The first search pass was rejected after its query generator produced invalid test cases. That complete pass was retained, the generator was corrected, and all 500 requests were run again. An apparent warm-refresh regression was also retained and traced to switching resource profiles before the true same-profile refresh was measured.

A benchmark that cannot fail is a marketing prop. Rejected and inconvenient observations remain part of this record.

Locked provenance

The raw evidence bundle contains the machine result, timing record, source-candidate patch, database validation, corrected and rejected query distributions, targeted tool measurements, refresh records, and host diagnostics. Every artifact is covered by a retained SHA-256 manifest.

Evidence manifest

1e1e6cfaa04d4d4db54597bb42766272e4d6a803dc5c372c5ee9f45dce042520

Candidate source patch

c9ec223886695bfbb25b363ede5c76539eae668e6d3d20c1bf8b05424eadb6b4

Semantic projection

e2282fd551592a3d1e4b89e8940ef21b2f1b615e5bc0eb432572cb3cad90b354

Database integrity

quick_check = ok

The raw bundle remains private because it contains absolute host paths. This page publishes the redacted measurements and the hashes that bind them to the retained record.

A different kind of multi-million-symbol repository

The checkout contained 94,743 discovered files at Linux revision 248951ddc14de84de3910f9b13f51491a8cd91df. Both versions produced 4,782,846 symbols, 84,107 relationships, and 86 semantic anchors.

Preview 7 removed 22.32 seconds from relationship resolution, but added 82.45 seconds to extraction and commit. Unlike Firefox, this corpus did not have a multi-hour resolution tail, so the cost of building the compact retrieval index is plainly visible.

Multi-second scans become interactive lookups

The matched workload used 25 distinct queries in each of five classes, four passes, deterministic rotating interleave, and one persistent MCP process per version. All 1,000 measured requests completed without tool errors.

The speedup did not come from simply returning less. Preview 7 repeatedly found useful candidates for the broad concepts virtual filesystem inode and system call dispatch, where Preview 6 returned nothing. A third probe,cgroup resource controller, remained empty in both versions. This is a repeatable sample, not a complete relevance study.

Existing graph navigation remained fast: a 20-item outline measured 1.76ms p50 and a populated reference lookup 3.10ms p50 in Preview 7. The complete 500-request workload sustained 61.44 sequential requests per second, compared with 0.49 in Preview 6.

Stored meaning matched; product boundaries remained visible

Complete canonical projections were generated independently from both databases and hashed. All 77,865 indexed files, 4,782,846 symbols, 84,107 relationships, and 86 semantic anchors matched exactly. Both databases returned PRAGMA quick_check = ok.

The costs

• Fresh Full indexing is 46.1% slower on this corpus.
• The database grows by 274.1 MiB, or 8.97%.
• Extraction and database writing are roughly 81–82% slower while the compact retrieval structures are built.
• Starting a fresh MCP process rises from about 3.26 seconds to 4.67 seconds, making a persistent session important.

The coverage boundary

Atlas Scout currently extracts 4.61 million C/C++ symbols from this checkout, but its C/C++ extractor is partial and this database contains no C/C++ relationships. The Rust and Python relationship graph is populated. This benchmark validates search across a multi-million-symbol kernel corpus; it does not claim a complete Linux C call graph.

Linux method and locked provenance

This is a same-host, same-corpus, fresh-database comparison—not a reboot-cold test. Each version received two fresh Full indexes in the alternating order Preview 6, Preview 7, Preview 7, Preview 6. The midpoint is the arithmetic midpoint of two close observations, not a high-confidence population median.

Search measurements came from persistent, sequential MCP sessions against the completed databases. One warm-up was excluded and percentiles use nearest-rank observations. Preview 6 was the public release binary; Preview 7 was the same recorded local source candidate used for the Firefox test.

Evidence manifest

eed3b1a7c223425008b18099f622889d7ae9df03a542432597b4b9312c7e03a8

Candidate source patch

c9ec223886695bfbb25b363ede5c76539eae668e6d3d20c1bf8b05424eadb6b4

Symbol projection

2812dd02ae96126e8d953ce2dde0652630ed7f891eaf10f056ba031052ad1718

Database integrity

quick_check = ok

The raw evidence bundle remains private because it contains absolute host paths. The retained manifest covers all 30 evidence files; this page publishes redacted results and the hashes that bind them to that record.