Benchmarks
Reproducible performance and quality measurements from repositories large enough to expose where Atlas Scout is wrong.
MCP context · Preview 14
Quiet when healthy. Explicit when judgment needs help.
Context is the scarcest resource an agent has, and Atlas Scout was spending it on reassurance. Every routine exact result carried the same 635-byte health and capability envelope - even when the index was fresh, the language fully supported, and the page complete. Preview 14 makes absence meaningful: the normal case carries an empty _meta object, while exceptions keep the diagnostics that protect an agent from over-trusting the answer.
Less catalog material before the first useful answer
| Measured compact-JSON surface | Before | Preview 14 | Reduction |
|---|---|---|---|
Pro tools/list, 14 tools | 211,335 B | 115,889 B | 45.2% |
Free tools/list, 6 tools | 91,933 B | 51,611 B | 43.9% |
| Fourteen output schemas | 197,421 B | 101,736 B | 48.5% |
Healthy exact-result _meta | 635 B | 2 B ({}) | 99.7% |
The useful information stays conditional
Empty results retain trust, health, and relevant language support. Partial languages retain capability warnings. Indexing, stale, degraded, failed, or shared-writer states retain health. Truncated pages retain measurements and guidance; repaired arguments retain their repair ledger; symbol_schema still returns the full census.
Connected evidence also remains available, but it is requested deliberately. In the baseline audit it added 453 bytes to a one-hit exact search and 6,740 bytes to a five-hit broad search. Preview 14 guides agents to ask for that neighborhood only when it is likely to replace a separate structural call.
What these numbers do - and do not - claim
The catalog reduction primarily affects startup and compatibility material. Some hosts strip output schemas before model invocation, and stable tool definitions may be prompt cached. The dynamic metadata reduction is the clearer recurring context improvement because each result otherwise introduces new conversation content.
These are byte measurements, not a claim that every host saves the same number of model tokens. No tool or evidence capability was removed. The measured product behavior was prepared in the immutable Preview 13 candidate; released Preview 14 carries it unchanged after a Windows line-ending acceptance repair outside these measured paths. Preview 14 passed 376 workspace tests, four analyzer tests, the VS Code compile, and version-sync verification; five live-service tests remained intentionally ignored.
Go · Preview 12 baseline / Preview 14
Package evidence replaces name coincidence
A shared name is a coincidence, not a connection. Preview 12 could connect a Go call to a globally unique function or method with the same terminal name - even across package and language boundaries. Preview 14 instead resolves only inside a proven source package, exact indexed import, or static receiver type. When that evidence is missing, the call remains usefully unresolved.
Where the two resolvers disagreed on shared observations
| Shared-observation disposition | Calls | Meaning |
|---|---|---|
| Both resolved to the same target | 12,224 | Agreement |
| Candidate-only target | 5,595 | Package- or type-proven |
| Preview 12-only target | 6,630 | Global-name evidence declined |
| Neither resolved | 61,859 | No proven indexed target |
| Different resolved targets | 0 | No conflicting target choices |
Preview 12 produced 83 cross-language Go targets on Ollama, including Go calls attached to TSX symbols through name collisions. Preview 14 produced zero. Standard-library selectors such as fmt.Println, json.Unmarshal, and slog.New also stayed unresolved instead of borrowing an unrelated workspace symbol.
The measured cost
| Ollama metric | Preview 12 | Preview 14 | Change |
|---|---|---|---|
| Median fresh-index wall time | 2.165s | 2.825s | 30.5% slower |
| Peak RSS | 170,786 KiB | 175,758 KiB | 2.9% higher |
| Database size | 55,611,392 B | 57,753,600 B | 2.04 MiB larger |
| No-change refresh | 6ms | 6ms | Unchanged |
| One-file Go refresh | 0.34–0.35s | 0.79–0.80s | About 0.45s slower |
Exact and broad symbol-search distributions remained within a few milliseconds of the Preview 12 baseline. The obvious optimization target is incremental invalidation: Preview 14 currently reconsiders unresolved Go calls after a declaration changes.
What Preview 14 supports
The implemented subset covers same-package bare calls, exact indexed imports - including aliases and local replacements - and direct methods on proven receivers such as declared method receivers, typed parameters or locals, imported named types, and unambiguous composite literals. Test packages, nested modules, lexical import shadowing, and fail-closed build variants are part of the evidence.
This is not compiler-complete Go analysis. Unproven receiver expressions, promoted and embedded methods, complete pointer/value method sets, dot imports, method expressions and values, factory inference, full build-context evaluation, and cgo calls remain unresolved or future work. Dependencies are never downloaded or fabricated.
Method and publication boundary
Ollama was pinned at commit 83d4311ffecb79bf3a7a1f0341afa4eb069da853. Two immutable optimized executables were alternated against fresh external caches on the same worktree. The 996 supported files completed with zero failures. These are warm filesystem measurements, not reboot-cold measurements.
The measurements were recorded before promotion against the product behavior shipped in Preview 14. Preview 13 failed an unrelated Windows line-ending acceptance boundary and remains immutable; the repair does not change these measured paths.
C++ · definitions-only control / Preview 14
A useful graph built from proof, not plausibility
Preview 12 extracted C++ definitions but no C++ relationships. Preview 14 retains bare, qualified, dot-member, and arrow-member call sites, then resolves only through exact lexical ownership, simple proven receiver types, and same-file or directly resolved include visibility.
Kodi produced 226,661 C++ call observations. Preview 14 resolved 8.977% and deliberately left 206,314 unresolved. That refusal is the headline: every edge in the graph is one Scout could prove. A maximum of 25 candidates appeared for one call, but the 99th percentile remained two despite nearly 70,000 callable symbols.
| Observed form | Observed | Resolved | Initial evidence lane |
|---|---|---|---|
| Bare | 61,492 | 18,976 | Enclosing owner, then exact lexical scope |
Qualified :: | 44,000 | 394 | Exact qualified target |
| Dot member | 71,394 | 653 | Simple statically declared receiver |
| Arrow member | 49,775 | 324 | Simple pointee or this type |
Repairing the targets before adding edges
The old definition population contained 4,472 callable symbols nested under another callable - at least 2,496 were ordinary direct-initialization variables named lock. Preview 14 reduced that population to 18, a 99.60% reduction, and excluded all remaining 18 from the resolver because preprocessor recovery still left their ownership uncertain.
Corpus-wide audits found zero C++ calls targeting another language, zero cross-file targets outside direct resolved-include visibility, zero targets nested under a callable, and zero failed files. External-library calls such as std::move, std::make_shared, and fmt::format remain observations rather than borrowed workspace symbols.
The measured cost of the C++ semantic graph
| Kodi metric | Pre-C++ control | Preview 14 | Change |
|---|---|---|---|
| Fresh-index midpoint | 7.121s median | 11.274s midpoint | 58.3% slower |
| Peak RSS | ≈230 MiB | ≈400 MiB | ≈74.6% higher |
| Database size | 117,440,512 B | 295,739,392 B | 2.518× |
| Exact standalone search p50 | 185ms | 328ms | 77.3% slower |
| No-change internal path | - | 47ms | Preview 14 record |
| One-file update | - | 1.416s internal | Preview 14 record |
The database adds about 170 MiB. Standalone search pays visible process and database-open cost, while one-file updates still reconsider the unresolved C++ population. Storage, include representation, and incremental invalidation are explicit optimization targets.
What remains outside the claim
The initial subset does not claim compiler equivalence, transitive include awareness, template-aware overload dispatch, complete receiver inference, function-pointer or callable-object resolution, virtual dispatch, build-variant selection, or external library modeling. Only 163 of 30,859 include observations resolve to an exact indexed file in the first lane.
The pinned Kodi checkout contained 6,337 supported files and more than one million C/C++ lines. It indexed with zero failures, produced deterministic Preview 14 databases, passed database integrity checks, and accompanied a complete workspace suite of 374 passed, 0 failed, and 5 ignored live-service tests.
The measurements were recorded before promotion against the product behavior shipped in Preview 14. Preview 13 failed an unrelated Windows line-ending acceptance boundary and remains immutable; the repair does not change these measured paths. C++ remains partial: the release claim is a bounded, high-confidence call subset rather than a complete C++ compiler model.
Rust · Preview 10 baseline / Preview 12 source candidate
Qualified calls, with proof or no edge
Preview 12 can follow qualified Rust calls through proven owner, module, import, re-export, dependency, and exact UFCS boundaries. If the indexed Cargo and source evidence cannot prove one target, Scout leaves the call unresolved instead of falling back to a same-name method somewhere else.
The advertised regression subset contained 38 observed qualified calls: 28 resolved to the expected definition and 10 deliberately stayed unresolved. Two calls on the same source line remained separately addressable at byte offsets 610 and 630.
Fresh-index change from Preview 10
| Pinned corpus | Preview 10 | Preview 12 | Index-time change | Database change |
|---|---|---|---|---|
| Firefox | 458.252s | 390.560s | 14.77% faster | 2.96% larger |
| Linux kernel | 262.208s | 210.868s | 19.58% faster | 0.20% larger |
| Tokio | 1.188s | 1.354s | 13.97% slower (+166ms) | 7.11% larger |
The large-corpus wins and small-corpus loss are both part of the result. Preview 12 does more one-time Rust extraction work. That investment removed 26.185 billion candidate comparisons on the Firefox run, while its fixed cost remained visible on Tokio.
Exact source sites survive the MCP boundary
SymbolStore::new returned all 19 observed incoming sites in two complete pages of 10 and 9 observations. The measured pages took 5.345ms and 6.341ms. On the Firefox index, warm incoming-reference and graph queries had medians of 3.973ms and 3.837ms; edit_impact remained an honest 1,083.207ms outlier.
Evidence boundaries
- The Linux checkout produced 6,219 qualified Rust observations and zero resolved qualified targets because that benchmark had no indexed Cargo project graph. Scout emitted no candidates or guesses.
- The 100% precision and recall result applies to the 38-case advertised regression subset, not every possible Rust program.
- Scout is not a Rust compiler. Glob-derived visibility, missing dependencies, macros, generated modules, and unproven trait or expression-type selection remain outside the proven resolution lanes.
- Corpus pins, host state, cache protocol, command lines, raw records, and checksums are retained with the official evidence rather than inferred from these summary figures.
Architecture query · Preview 12 regression record
A bounded answer now has bounded work
The worst kind of slow is the kind that keeps running after you have given up. A response limit constrained the final architecture summary, but not the database work used to build it: on a 398-file index, the old query shape timed out at the client after five minutes - and then kept consuming a CPU core for more than 28 minutes.
Preview 12 pre-aggregates symbol and resolved-edge counts by file before joining the bounded module view. The same database measured 0.01 seconds for module aggregation and 0.01 seconds for edge aggregation after the query-shape correction.
The SQLite work now runs outside the async runtime, observes cooperative cancellation, and carries a five-second internal budget so an abandoned client cannot leave the same unbounded query running in the background.
This is a regression record for one pathological SQL shape on one pinned 398-file database, not a universal end-to-end speed claim. The important product guarantee is containment: response bounds, cancellation, and database-work bounds now agree.
Two repositories, two lessons
Firefox and Linux stress different parts of Atlas Scout. Firefox exposed a resolver that did far too much work before the index became ready. Linux exposed the cost - and the payoff - of building a reusable search path for the questions an agent asks after indexing.
These are separate same-host measurements with different protocols. The Firefox result is the stronger end-to-end scaling test; Linux is the stronger controlled search tradeoff comparison.
Firefox · reboot-cold Preview 7 candidate
From two and a half hours to four minutes
We do not merely benchmark Atlas Scout to prove it is fast. We benchmark it to discover where it is wrong.
Preview 6 indexed the Firefox checkout correctly - and then kept you waiting. Complete fresh readiness took 2 hours, 32 minutes, and 46 seconds, and 98.34% of that time went to a single phase: resolving relationships across files. The map was right. It just arrived too late to be part of a working session.
The Preview 7 source candidate indexed the same checkout on the same machine, starting from a rebooted host and a verified-empty cache, in 4 minutes and 0.43 seconds: 38.1× faster end to end. The resolution phase that had consumed the afternoon fell from more than two and a half hours to 15.823 seconds, a 569.7× improvement. The wait for a Firefox-scale structural map went from the length of a feature film to the length of a coffee break.
Complete fresh-index time
What those four minutes bought
Not a sample, and not a shortcut. Four minutes produced a complete structural map of the entire supported Firefox checkout:
The checkout contained 469,704 discovered files at Firefox revision f4e6e71e9c4deb3818880f2c8a22de09a92955ce. Atlas Scout indexed every supported file with zero failed files.
Where the indexing time went
Then the questions begin
An index is only worth what it answers afterwards. This is the part of the story an agent lives in: not the four-minute build, but the hundreds of questions that follow it. The corrected search benchmark measured 500 sequential requests in one persistent MCP session: 25 distinct queries across five classes, repeated in four interleaved passes. Initialization, readiness checks, and one warm-up were excluded. All requests completed without tool errors.
The answers came back at conversation speed. An exact scoped lookup returned in about a millisecond, and even a broad fuzzy concept search stayed near 63 milliseconds.
Median query latency at Firefox scale
| Query class | p50 | p95 | p99 |
|---|---|---|---|
| Exact unique name | 1.18ms | 1.69ms | 1.78ms |
| Ambiguous exact name | 1.48ms | 2.62ms | 11.47ms |
| Path-scoped exact name | 1.09ms | 1.70ms | 9.01ms |
| Broad fuzzy concept | 63.01ms | 133.99ms | 161.82ms |
| Exact miss | 1.68ms | 1.99ms | 2.04ms |
A separate 300-request run measured a populated reference lookup at 2.92ms p50 and a 20-item file outline at 3.41ms p50. These timings include serializing useful model context, not only database lookup time.
Faster without giving the model less
Speed bought by guessing would be worthless. The optimization changes how Scout narrows possible imported targets, not the rule for accepting a relationship. Preview 6 and the Preview 7 candidate produced the same 1,204 import-binding and 2,378 import-path resolutions. Ambiguous imports remain unresolved rather than guessed.
The ordered projection of relationship identity, resolved target, strategy, and confidence reproduced the expected SHA-256:
e2282fd551592a3d1e4b89e8940ef21b2f1b615e5bc0eb432572cb3cad90b354Scoped searches also passed deliberately difficult probes: a common function named test was returned from the requested file, while a scoped search for Distance still found compound CamelCase names. The final database passed its integrity check.
The costs are part of the result
There is no free 38×. Preview 7 deliberately spends more during indexing so repeated agent queries do less, and those costs belong in the record:
- The generated database is 5.15% larger.
- Peak memory during the reboot-cold run is 4.0% higher.
- Extraction and commit take 49.1% longer before the much shorter resolution phase.
- Starting a fresh MCP process still takes about 5.7 seconds on this machine.
- A complete workspace schema summary remains roughly a 346ms operation.
- A same-profile no-change refresh took 8.32 seconds of process wall time, compared with 7.07 seconds in Preview 6.
The accepted trade is modestly more local storage and indexing work in exchange for removing the multi-hour readiness tail and making repeated navigation interactive.
Method and interpretation boundaries
Both completed databases used the same Firefox checkout and the same Linux workstation: an AMD Ryzen 9 5950X with 32 GiB of memory and a Samsung 980 PRO NVMe SSD. The Preview 7 command began roughly three minutes after reboot. No recursive operation had traversed the source tree, and the external Atlas Scout cache was new and verified empty.
The Preview 6 control used a fresh database on the same host and corpus, but it did not begin after reboot. Preview 6 used the public release binary; Preview 7 used a recorded local source candidate built with Rust 1.95.0, while release CI uses Rust 1.94.0. The measurements therefore validate the source candidate, not the byte identity or a universal performance guarantee for the eventual public artifact.
Query percentiles are nearest-rank observations from warm persistent MCP sessions. MCP initialization, readiness, and one warm-up were excluded from those distributions and reported separately.
The benchmark was tested too
The first search pass was rejected after its query generator produced invalid test cases. That complete pass was retained, the generator was corrected, and all 500 requests were run again. An apparent warm-refresh regression was also retained and traced to switching resource profiles before the true same-profile refresh was measured.
A benchmark that cannot fail is a marketing prop. Rejected and inconvenient observations remain part of this record.
Locked provenance
The raw evidence bundle contains the machine result, timing record, source-candidate patch, database validation, corrected and rejected query distributions, targeted tool measurements, refresh records, and host diagnostics. Every artifact is covered by a retained SHA-256 manifest.
- Evidence manifest
1e1e6cfaa04d4d4db54597bb42766272e4d6a803dc5c372c5ee9f45dce042520- Candidate source patch
c9ec223886695bfbb25b363ede5c76539eae668e6d3d20c1bf8b05424eadb6b4- Semantic projection
e2282fd551592a3d1e4b89e8940ef21b2f1b615e5bc0eb432572cb3cad90b354- Database integrity
quick_check = ok
The raw bundle remains private because it contains absolute host paths. This page publishes the redacted measurements and the hashes that bind them to the retained record.
Linux kernel · controlled Preview 6 / Preview 7 comparison
Index once. Ask many questions.
This story opens with a loss. Preview 7 took a two-run midpoint of 3 minutes and 10.94 seconds to build a fresh Full index of the Linux kernel, compared with 2 minutes and 10.69 seconds in Preview 6. That is a real 46.1% regression, and the database is 8.97% larger. Both numbers lead this section on purpose.
Then the investment starts paying rent. A matched 500-request search workload against the completed index fell from 16 minutes and 50.14 seconds to 8.14 seconds: 124.1× faster. For this deliberately equal mix of five query classes, Preview 7 earned back its extra indexing minute after approximately 30 requests.
When the indexing investment is recovered
A different kind of multi-million-symbol repository
The checkout contained 94,743 discovered files at Linux revision 248951ddc14de84de3910f9b13f51491a8cd91df. Both versions produced 4,782,846 symbols, 84,107 relationships, and 86 semantic anchors.
Preview 7 removed 22.32 seconds from relationship resolution, but added 82.45 seconds to extraction and commit. Unlike Firefox, this corpus did not have a multi-hour resolution tail, so the cost of building the compact retrieval index is plainly visible.
Multi-second scans become interactive lookups
The matched workload used 25 distinct queries in each of five classes, four passes, deterministic rotating interleave, and one persistent MCP process per version. All 1,000 measured requests completed without tool errors.
The headline: finding one exact symbol among 4.78 million fell from 3.69 seconds to under a millisecond - the answer now arrives faster than the keystroke that asked for it.
Median search latency across 4.78 million symbols
| Query class | Preview 6 p50 | Preview 7 p50 | Improvement |
|---|---|---|---|
| Exact unique name | 3,688.19ms | 0.99ms | 3,714.8× |
| Ambiguous exact name | 891.14ms | 1.98ms | 450.8× |
| Path-scoped exact name | 892.26ms | 1.29ms | 692.5× |
| Broad fuzzy concept | 3,031.12ms | 60.73ms | 49.9× |
| Exact miss | 757.55ms | 1.32ms | 574.8× |
The speedup did not come from simply returning less. Preview 7 repeatedly found useful candidates for the broad concepts virtual filesystem inode and system call dispatch, where Preview 6 returned nothing. A third probe,cgroup resource controller, remained empty in both versions. This is a repeatable sample, not a complete relevance study.
Existing graph navigation remained fast: a 20-item outline measured 1.76ms p50 and a populated reference lookup 3.10ms p50 in Preview 7. The complete 500-request workload sustained 61.44 sequential requests per second, compared with 0.49 in Preview 6.
Stored meaning matched; product boundaries remained visible
A speedup that changed the answers would be a bug, not a feature. Complete canonical projections were generated independently from both databases and hashed. All 77,865 indexed files, 4,782,846 symbols, 84,107 relationships, and 86 semantic anchors matched exactly. Both databases returned PRAGMA quick_check = ok.
| Projection | Rows | Result |
|---|---|---|
| Indexed files | 77,865 | Identical |
| Symbols | 4,782,846 | Identical |
| Relationships | 84,107 | Identical |
| Semantic anchors | 86 | Identical |
The costs
- Fresh Full indexing is 46.1% slower on this corpus.
- The database grows by 274.1 MiB, or 8.97%.
- Extraction and database writing are roughly 81–82% slower while the compact retrieval structures are built.
- Starting a fresh MCP process rises from about 3.26 seconds to 4.67 seconds, making a persistent session important.
The coverage boundary
Atlas Scout currently extracts 4.61 million C/C++ symbols from this checkout, but its C/C++ extractor is partial and this database contains no C/C++ relationships. The Rust and Python relationship graph is populated. This benchmark validates search across a multi-million-symbol kernel corpus; it does not claim a complete Linux C call graph.
Linux method and locked provenance
This is a same-host, same-corpus, fresh-database comparison - not a reboot-cold test. Each version received two fresh Full indexes in the alternating order Preview 6, Preview 7, Preview 7, Preview 6. The midpoint is the arithmetic midpoint of two close observations, not a high-confidence population median.
Search measurements came from persistent, sequential MCP sessions against the completed databases. One warm-up was excluded and percentiles use nearest-rank observations. Preview 6 was the public release binary; Preview 7 was the same recorded local source candidate used for the Firefox test.
- Evidence manifest
eed3b1a7c223425008b18099f622889d7ae9df03a542432597b4b9312c7e03a8- Candidate source patch
c9ec223886695bfbb25b363ede5c76539eae668e6d3d20c1bf8b05424eadb6b4- Symbol projection
2812dd02ae96126e8d953ce2dde0652630ed7f891eaf10f056ba031052ad1718- Database integrity
quick_check = ok
The raw evidence bundle remains private because it contains absolute host paths. The retained manifest covers all 30 evidence files; this page publishes redacted results and the hashes that bind them to that record.