What we measured, and what it refuted

The first three are the pitch. The fourth is the trade we made on purpose and would make again — quality parity was never claimed. The fifth is the one real defect these benchmarks found — a settings value, not an architecture, since raised in production. The sixth is the measured limit of the whole recall-recovery story: leCore's ranker is lexical, and at exactly zero question–answer overlap no top_k recovers it (§08).

A FAISS index does not exist until an embedding model has read every document. Latency-only comparisons skip that step. This measures model load + index build + one query — what a user actually waits for on a cold corpus.

Shaw asked us to stop quoting our own test and run the standard one. We did, then ran three real embedding models through the same harness so it isn't ours-versus-literature. bge-base beats leCore on all four tasks. "BM25 is a floor" was correct.

Scored three ways, never two: HIT, MISS, and UNRUNNABLE — the endpoint refused the request. Collapsing a refusal into a wrong answer would let a memory system claim it "beat" a model that was never allowed to compete.

Cells answered correctly, 900k–5M tokens, n=6 per tier. Beyond 1M the control returns "The input token count exceeds the maximum number of tokens allowed." Tokens the model reads stay flat: a 60k corpus → ~1,458 read; a 5M corpus → 3,401 read.

Two identical 303M language models — same seed, same schedule, same step count, RoPE in both arms; only the attention operator differs. We published this as 41.5% worse for HRR. Then we re-ran it against our own methodology, and 41.5% became 32.7%. About a fifth of our headline negative was our configuration, not the operator. The verdict is unchanged — the operator still loses — but the number we shipped was inflated, so here is the corrected one.

Sources: the arms' own final JSONs — results/runpod_303m_rescue/final_softmax.json (3.149412655830383) and the HRR arm's out/final_hrr.json on its Vast box, mirrored to results/hrr_vs_softmax/303m_hrr_vast/ (4.180486848950386). Sweep logs: sweeps303/sweep_hrr_*.log, six files, one per rate. The superseded pair is 303m-results/final_softmax.json and is kept, not deleted. Checkpoints pushed to staccs/lecore-303m-rerun.

Cotten asked for this twice: plant decoys that carry the full query vocabulary but are not the answer. He was right — at the shipped top_k=8 it breaks, and that turned out to be a settings value rather than an architecture problem.

• "the vault access code was rotated last quarter"
• "do not share the vault access code with contractors"
• "Archived: previous vault access code was MALLARD-0000-TULIP" — the nastiest: a code-shaped wrong answer

The control row is the whole finding

On the zero-decoy corpus a dumb exact-substring counter — no idf, no length normalisation, no saturation — scores precision@1 = 1.00, identical to leCore. A benchmark that a substring counter aces is not measuring a ranker. It is measuring the corpus's lexical confusability, and ours was ~zero.

…and the fix is to buy more context, which is the entire pitch

We were never trying to out-rank bge-base. So the right answer to "the needle fell to rank 12" is not a better ranker — it is a bigger top_k. Recall is fully recoverable at every decoy level measured, and the k required is roughly the number of competing passages.

top_k needed to bring recall back to 1.0 — swept over 8, 12, 16, 24, 32, 48, 64

What does survive

leCore's BM25 is genuinely ranking rather than guessing. tied@top is ≈1.0 for leCore against 5–85 for the substring floor, so the floor's occasional wins are argsort index luck while leCore's are real separation. Clustering also helps, because a clustered decoy lands inside the answer's own chunk instead of competing with it as a separate document.

This bench exists because a critic asked. Cotten, publicly: "rewrite the benchmark so it never skips to BM25; always profile the VSA over 1024d and pay attention to _Index — btw the _Index wrapper isn't optimized." So the pure VSA path is now a first-class, never-skipped row, and every number ships regardless of direction. He was right about the wrapper. And the same profile handed the VSA — the arm §02 removed from the hybrid — its first outright win, on the exact adversarial bench that broke BM25 in §05.

The win: decoy resistance

Same protocol as §05 — 800 lines, 32 seeds per cell, k=8, scattered decoys the hard case. The idf-weighted dense superposition holds rank under exactly the term-collision that inverts BM25's scoring.

The honest read of the win: at 10+ scattered decoys the VSA's precision@1 collapses too — 0.09 → 0.03. What it keeps is recall: at 25 scattered decoys the answer is still inside a k=8 context 69% of the time against BM25's 22%, before any of §05's k-raising. Clustered decoys cost it nothing at all: P@1 = 1.00 at 2, 10 and 25, where BM25 falls 1.00 → 0.72 → 0.25.

What this leaves: three lanes, each measured where it wins

Method: local Apple M5 Pro (18 cores), Python 3.14.6 / numpy 2.5.0, $0 spent, wall clock 137 s. Load average was 5–6 throughout and is recorded per-section in results.json — a timing-contamination flag we publish rather than hide. Two thread arms, pinned single-thread and free, both reported; the ladder above uses the free arm (the single-thread floor was 3.7 ms, the single-thread wrapper 21.3 s — same shape). At 1k docs the auto index is exact and the wrapper is 2–4 ms; at 12k docs the forest kicks in and the wrapper is already 1.7–2.3 s. The rebuild dominates everywhere the forest exists.

§5's decoys were templated strings we wrote ourselves, which left one open question: does recall recovers with k survive corpora where the confusable passages are real? So: three BEIR corpora with real judged queries, leCore BM25 against GPU-embedded baselines in the same harness, hit@k swept from 1 to 200. The question is the crossover: at what k does leCore's hit rate reach what bge-base delivers at k=10? Answer: k = 32–64 on all three, at 3.2–6.4× the retrieved tokens. And below, this section's own open question re-run at scale: the same crossover at 2.68M documents lands at k = 128 — exactly the production cap.

Method and the honest fine print: clean Vast RTX 4090, load average 1.5, embedding on GPU. bge-HNSW matched exact flat search on every corpus — identical on SciFact and ArguAna, within 0.007 hit on NFCorpus — so there is no ANN recall loss to exploit at these sizes. And on this box the embedders' index build was as fast or faster than our BM25 build (bge 11.3–15.5 s GPU vs leCore 15.8–50.5 s CPU): the §01 cost-to-first-answer win assumes no GPU is present. One boundary on everything above: the ranker is lexical — §08 measures the case where a question shares zero vocabulary with its answer, and the k lever does not survive it.

At 2.68M documents, the crossover survives by a single k: it lands exactly on the cap

The three corpora above top out at 8,674 documents — §12's stated "not built" was whether crossover k grows with corpus size. It does. On Natural Questions — 2,681,468 docs, 3,452 judged queries, same harness, same crossover definition — bge-base's hit@10 is .781, leCore's is .475, and leCore does not reach .781 until k = 128 — exactly the production cap (hit@128 = .784, at 12.8× the tokens, ~14,142 tok/query). The trend across everything measured: 3.6–8.7k docs → k = 32–64; 2.68M docs → k = 128. Zero headroom left.

The fine print, all of it. The shipped BM25 build cannot run at this scale: its postings loop is O(vocab × N) — ~10¹² iterations at 2.68M docs — and does not terminate. The bench used an O(total tokens) rebuild whose postings are bit-identical (same idf, same per-(term,doc) weight expression, same ordering; scores() output identical). The 309.8 s is that rebuild, and the shipped build's non-termination is itself a reported finding — the fourth config-class engineering defect these benches have found, after top_k=8, the cap of 32, and §06's per-query forest rebuild. The box was not clean: RTX 4090, load average 6–24 across the arms, recorded per-arm in the JSON — timings carry that flag; hit rates are load-immune. e5-large was skipped at this scale (3× bge's encode cost for the same question), and only NQ has run — HotpotQA at 5.2M docs is the next tier and is not measured.

The loss, plainly: leCore's ranker is lexical. A question sharing zero vocabulary with its answer defeats it, and buying more context does not help. 64 needle/question pairs per overlap level, with the zero enforced in code, not by eyeballing — stemmed content-token sets must not intersect, no two content tokens may share a 4-character prefix, no shared numbers. Needles were embedded in ~222k tokens of real fineweb prose (~1.9k chunks), 3 positions × 2 seeds, through the deployed 600-char chunker. At measured Jaccard 0.000, leCore BM25 goes R@8 = .005 → R@64 = .023 — flat. Unlike §05 and §06, raising k recovers nothing, because there is no lexical anchor to rank on.

Method and the caveat to carry: pairs generated by Qwen2.5-7B-Instruct, then gate-enforced with rejects counted and published — at the zero level 65 rejected for stem overlap, 15 for shared prefixes, 8 for shared digits; 0 needles destroyed by chunking; 384 trials at J=0 and 378 at each other level. The no-shared-digit rule sometimes forces zero-overlap questions to alter quantities (a "50 new sensors" needle asked about as "30 extra devices"). That is fine for retrieval scoring — only chunk identity is scored — but the J=0 pairs are not quotable as QA pairs.

This gradient is not a synthetic artifact. §09 reproduces it on real code: paraphrased questions about a 47k-token repo retrieve perfectly, and the same question type collapses to .30–.40 on a 4.3M-token repo — while identifier-anchored questions stay the lexical ranker's case throughout. And the one system not flat at J=0 is now leCore's own lane: §10 quantizes bge-base with our machinery — recall@64 at J=0 identical to full bge — and prices exactly where it can afford to run.

The product's actual emerging user is an agent throwing a codebase at zoo_ask — bind the repo over MCP, then ask questions about it. This bench measures that path, factually, for the first time. Three Python repos at pinned tags — requests v2.32.3 (~47k tokens, 18 files), fastapi 0.115.12 with its tests (~625k, 360 files), django 5.1.7 (~4.3M, 2,107 files) — and 82 questions generated by Qwen2.5-7B against sampled ground-truth functions, gated by an answerable-from-chunk judge, in three types: identifier-anchored (must name the function), behavioral paraphrase (programmatically gated to contain no identifier from the code — the semantic stress), and cross-file ("what calls X and what happens to its return value"). Gold answers are (path, line-range) regions, so the production 600-char chunker and function-level chunking score against the same truth.

The claim shape this backs: identifier and cross-file questions are the lexical ranker's case, at zero embedding cost — cross-file asks go .90 at k=8 → 1.00 by k=64, identifier .73 → .85, and leCore indexes the 4.3M-token django tree in ~3.0 s on CPU where bge-base needs ~40–42 s of GPU embedding per chunking. Paraphrase questions need a semantic stage — and bge-base is not dominated here, stated plainly below.

Cross-file scored strictly — definition and caller both in the top-k — at k=64: leCore .77 (600-char) / .83 (function); bge .97 (600-char) but only .67 (function). Chunking is a real variable, not a footnote: function-level chunking helps bge on identifier asks (django h@8 .80 vs .40) and mildly helps leCore's paraphrase, while 600-char windows serve strict cross-file better under bge.

Method and every caveat: one Vast RTX 4090 (~55 min, $0.53 total across three boxes, two of which never ran), destroyed after. n = 6–10 questions per cell — single cells carry ~±0.15 noise, so quote the gradients and pooled rows, never one cell. All three repos are Python. Questions are one model's style (Qwen2.5-7B; the answerable gate passed ~60–80% of candidates). bge truncates chunks at 512 tokens — its real behavior — and function chunks were capped at 6,000 chars. The BM25 postings build was re-ordered doc-major to O(total tokens) — the same rebuild class §07's 2.68M-doc run needed — with score bit-identity vs the shipped build asserted on 323 chunks × 10 probes. Source: results/code_bench/ — questions.json with gold regions, results.json, summary.json, corpus_stats.json.

§08 ended at the cliff: at zero question–answer overlap only bge-base retrieves anything, and §06 named a quantized-bge semantic stage as the third lane — staged, not shipped, unmeasured. This is the measurement. We ran leCore's own requantize machinery (holographic_refactor.quantize_group, a budgeted per-tensor bit ladder with embedding cosine as the fidelity metric) over bge-base-en-v1.5, then made the result take every exam family it will face: MTEB before/after on four tasks — GPU and CPU — semantic NIAH before/after, FAISS flat + HNSW, §05's adversarial protocol with the quantized encoder as a fourth arm, and cost-to-first-answer against leCore BM25 on CPU only.

the artifact — budgeted requant, 1% encoder-fidelity budget · ladder 3/4/5/6/8 bits · group 64 · 126 s

The encoder resisted. Layers 0–1 took 3-bit wholesale (cosine ≥ .995); the budget then walked the ladder up through the middle of the stack, and 11 tensors — clustered in layers 7–9 — refused even 8-bit. The same machinery pointed at a 9B decoder (a separate, still-unfinished run) was landing whole tensors at ~3 bits/weight: encoder embedding geometry tolerates far less weight noise than decoder next-token loss. Uniform arms for calibration: 4-bit everywhere is 93.4 MB at SciFact .72842 — 98.4% of full, the sweet spot only if a packed runtime existed; 3-bit everywhere is 82.7 MB at .6922, visibly damaged. Shipped conclusion: the budgeted artifact. Home: huggingface.co/staccs/lecore-bge-assimilated.

Where it slots into the decoy bench

Same protocol as §05, 32 seeds per cell, k=8. It holds precision exactly where BM25 dies — 0.81 vs 0.19 at two scattered decoys, 0.72 vs 0.03 at 100 clustered — then collapses at 25+ scattered like everything else (P@1 0.00, R@8 0.59). And §06's VSA keeps its crown: at 25 scattered decoys the VSA's R@8 = 0.69 against the quantized encoder's 0.59 — the three lanes each still hold a bench where they are best.

What the lane costs, CPU only

Time-to-first-answer = index (and for the encoder, embed) + one query, 8 CPU threads, no GPU anywhere. The gap narrows with corpus size and is still 24.4× at 1M tokens. Query-side the encoder is fine — 27–31 ms batch-1 median at 4–8 threads (192-thread oversubscription regresses: 301 ms median, 1.6 s p90 — pin your threads). Real-corpus probes embedded 3.3–7.3 docs/s, putting full CPU embeds of the four BEIR corpora at an estimated 18–87 minutes each. And the caveat that kills a free lunch: requantize buys size, not CPU speed — the artifact stores dequantized fp32 snapped to the quantized grid, so it embeds at exactly normal bge-base CPU speed, and the 132 MB packed figure is an estimate, not a shipped file; no packed loader is built.

Method and fine print: two runs on a shared 192-core EPYC 9454 host, load average 10–18 throughout, recorded per-row and per-task in the JSONs — every timing carries that flag; every score is load-immune. Before/after arms swapped the quantized weights into the HF cache blob and restored the original from a backup after each arm (the swap is logged; no separate checksum chain was recorded). The Unicron MP filter itself did nothing, as predicted — 71 of 73 tensors heavy-tail passthrough, 2 guarded, 0 filtered — requantization is the lever. Sources: results/bge_assim/ and results/bge_matrix/.

Three people told us we were wrong about something. Read each against the actual claim: much cheaper, and usable where the alternative cannot run at all. A confirmed counterclaim is only a defeat if it contradicts something we asserted — several of these are true and were priced in from the start.

the two claims that survive intact are the only two we made

The adversarial bench is the one that genuinely landed, and only at the shipped top_k=8: a config bug, not an architecture defeat. The real-corpora shotgun (§07) then confirmed the recovery lever on BEIR data at k ≤ 64, and its 2.68M-doc rerun answered the corpus-size question: the crossover survives by a single grid point — k = 128, exactly the production cap — while the same run's cost fields put the embedder's amortization at tens-to-hundreds of queries. The semantic NIAH (§08) then drew the lever's hard boundary: it requires at least one shared content word, and at measured zero overlap raising k recovers nothing — while one word in four restores full recall at the deployed k, ahead of bge-base. That boundary was named in the changelog as the likeliest failure before it was run, and it is published either way. And the code bench (§09) took the whole picture to the product's real workload: on repositories, identifier and cross-file questions are the lexical lane's case at zero embedding cost, the paraphrase gradient reproduces §08's cliff on real code, and bge-base is not dominated — it wins paraphrase outright, which is exactly where the staged semantic lane points. §10 then built and measured that lane with our own machinery: within .002 nDCG of full bge on all four MTEB tasks, identical at the J=0 cliff, 343× → 24.4× behind leCore on CPU cost-to-first-answer from 50k to 1M tokens — the trade priced end to end, the union negative published, and the stage shipped env-gated, default off. The newest row is the one nobody asked for: we re-ran our own 303M negative with our five confounds removed and found we had overstated it — 41.5% is 32.7% (§04). The operator still loses, so the verdict is unchanged; the magnitude was partly our methodology, and correcting a number that was already against us is the same discipline as publishing it in the first place.