We are not trying to be better. The claim is much cheaper and usable at context lengths where the alternative cannot run at all. Below is everything three people asked us to measure against that claim — including the results that went against us.
Every number is reproducible from github.com/staccDOTsol/supercontext. Model: gemini-2.5-flash via OpenRouter. Single runs at temperature 0.
The first three are the pitch. The fourth is the trade we made on purpose and would make again — quality parity was never claimed. The fifth is the one real defect these benchmarks found — a settings value, not an architecture, since raised in production. The sixth is the measured limit of the whole recall-recovery story: leCore's ranker is lexical, and at exactly zero question–answer overlap no top_k recovers it (§08).
A FAISS index does not exist until an embedding model has read every document. Latency-only comparisons skip that step. This measures model load + index build + one query — what a user actually waits for on a cold corpus.
HNSW makes queries sub-linear, exactly as Shaw said — but it does not touch the dominant term. Graph construction is ~0.3 s on top of ~182 s of embedding.
| corpus | docs | leCore index | flat index | HNSW index | leCore query | HNSW query |
|---|---|---|---|---|---|---|
| 50k | 597 | 0.05 s | 10.60 s | 10.60 s | 2.39 ms | 31.23 ms |
| 200k | 2,388 | 0.57 s | 42.38 s | 42.41 s | 0.86 ms | 31.03 ms |
| 1M | 11,940 | 13.31 s | 182.03 s | 182.28 s | 4.86 ms | 35.83 ms |
Shaw asked us to stop quoting our own test and run the standard one. We did, then ran three real embedding models through the same harness so it isn't ours-versus-literature. bge-base beats leCore on all four tasks. "BM25 is a floor" was correct.
| system | SciFact | ArguAna | NFCorpus | SCIDOCS | FiQA |
|---|---|---|---|---|---|
| leCore BM25 + expand | 0.6705 | 0.4314 | 0.3167 | 0.1577 | 0.2378 |
| all-MiniLM-L6-v2 | 0.6451 | 0.5017 | 0.3159 | 0.2164 | — |
| e5-large-v2 | 0.7221 | 0.4642 | 0.3715 | 0.2050 | — |
| bge-base-en-v1.5 | 0.7404 | 0.6375 | 0.3735 | 0.2172 | — |
Our own dense VSA encoder scored 0.4160 on SciFact and made the hybrid worse at every fusion weight (0.05 → .6596, 0.15 → .6307, 0.30 → .6057, against .6705 for BM25 alone). We were shipping the worst of three configurations. It has been removed.
Scored three ways, never two: HIT, MISS, and UNRUNNABLE — the endpoint refused the request. Collapsing a refusal into a wrong answer would let a memory system claim it "beat" a model that was never allowed to compete.
Beyond 1M the control returns "The input token count exceeds the maximum number of tokens allowed." Tokens the model reads stay flat: a 60k corpus → ~1,458 read; a 5M corpus → 3,401 read.
Two identical 303M language models on 5,000,002,551 tokens — same seed, same schedule, same step count, RoPE in both arms. Only the attention operator differs. This answers a question the project had left open twice, and the answer is no.
Both arms are undertrained (1B tokens ≈ 1/6 Chinchilla-optimal); no hyperparameter tuning — the learning rate was chosen for softmax; and torch.compile was on for softmax and off for HRR, which doesn't change the math but is an asymmetry we introduced. A rerun fixing all three is in progress.
Plant "The secret code is XXXX" at start/middle/end of filler and score the logit margin of the true code token against a random wrong one — >0 means the needle won. 3 seeds per cell, both arms RoPE, training block 1024. Softmax retrieves inside its training length, decays at 2048, and is dead by 4096–8192. HRR never localizes at any length. Its lone positive cells (+0.16 at 8192) are position-insensitive — near-identical at start, middle and end (0.159 / 0.159 / 0.151) — a uniform bias toward the answer token, not retrieval. At 8192 the honest read is both arms fail.
| context | softmax margin · start / mid / end | HRR margin · start / mid / end |
|---|---|---|
| 512 | +4.84 +4.82 +5.79 | −0.05 −0.05 +0.06 |
| 1024 | +1.33 +4.85 +4.71 | −0.14 −0.14 −0.10 |
| 2048 | +0.13 +1.72 +0.47 | −0.28 −0.28 −0.28 |
| 4096 | +0.13 +0.07 +0.08 | −0.59 −0.59 −0.60 |
| 8192 | −0.31 −0.30 −0.37 | +0.16 +0.16 +0.15 ← bias, not retrieval |
Mandatory caveat: the pair is not loss-matched — hrr val 4.5138 vs softmax 3.1902, 41.5% worse — so this confounds the operator with plain LM quality: a worse language model fails NIAH for reasons that have nothing to do with attention. A matched-loss rerun is in flight, and the caveats box above applies in full.
Cotten asked for this twice: plant decoys that carry the full query vocabulary but are not the answer. He was right — at the shipped top_k=8 it breaks, and that turned out to be a settings value rather than an architecture problem.
On the zero-decoy corpus a dumb exact-substring counter — no idf, no length normalisation, no saturation — scores precision@1 = 1.00, identical to leCore. A benchmark that a substring counter aces is not measuring a ranker. It is measuring the corpus's lexical confusability, and ours was ~zero.
Precision@1 dies at two decoys — 1.00 → 0.19. The surviving envelope is recall, not precision: the answer still reaches the model's context up to ~10 competing passages (recall@8 = 0.91) and breaks between 10 and 25 (0.22). Past that, at this k, the answer is not in the context at all — so no amount of model quality recovers it, and only a bigger top_k does. Mean margin goes negative (−0.44 at 25 decoys): the best decoy outscores the needle.
We were never trying to out-rank bge-base. So the right answer to "the needle fell to rank 12" is not a better ranker — it is a bigger top_k. Recall is fully recoverable at every decoy level measured, and the k required is roughly the number of competing passages.
Raising k from 8 to 64 is 8× the retrieved tokens — 68,057 → ~544k on the 60k–500k sweep — which is still 12.3× fewer than the control's 6,676,718. Precision@1 was never the product. The product is that the answer is in the context at a price the alternative cannot match.
This is a scaling relationship, not a free lunch. Token cost grows with the number of confusable passages — a property of the query rather than of corpus size, but whether confusability grows with corpus size is not measured here, and it is the thing that would break this. That bench has since run on real corpora — §07: recall recovers within k ≤ 64 on all three BEIR sets. Corpus-size scaling beyond ~8.7k docs is still open. And §08 now marks the lever's hard boundary: k-recovery requires at least one shared content word — at measured zero question–answer overlap, recall stays flat no matter the k.
The shipped default was top_k=8 with a hard cap of 32 — both too low; the cap alone foreclosed the 100-decoy case. Fixed and deployed: default 16, cap 128, verified live.
leCore's BM25 is genuinely ranking rather than guessing. tied@top is ≈1.0 for leCore against 5–85 for the substring floor, so the floor's occasional wins are argsort index luck while leCore's are real separation. Clustering also helps, because a clustered decoy lands inside the answer's own chunk instead of competing with it as a separate document.
Our headline needle number was inflated by vocabulary isolation. A bag-of-words model has no representation for contains the answer versus is about the answer. So precision@1 is not something BM25 can be made to deliver — and it is not something we need. The adversarial bench did not kill the thesis; it killed the precision-retrieval thesis and revealed a viable cost/recall tradeoff in its place. The claim is recall@k, bought with a bigger top_k on the one axis we are cheap on — never precision@1.
One arm does hold precision under exactly these decoys: the pure VSA path, profiled next — §06 is its first outright win, published beside its losses.
This bench exists because a critic asked. Cotten, publicly: "rewrite the benchmark so it never skips to BM25; always profile the VSA over 1024d and pay attention to _Index — btw the _Index wrapper isn't optimized." So the pure VSA path is now a first-class, never-skipped row, and every number ships regardless of direction. He was right about the wrapper. And the same profile handed the VSA — the arm §02 removed from the hybrid — its first outright win, on the exact adversarial bench that broke BM25 in §05.
The math is milliseconds. The shipped wrapper rebuilt the HoloForest on every query — the one-time forest build is 19.6–20.4 s, essentially the whole 20.5 s, ~4,200× the floor. Cotten's 518 ms sits 40× below our deployed number: his complaint was charitable. The fix is a cache — build the forest once at bind — and is shipping now; 20.5 s stands as the deployed number until the cached path is re-measured. Encoder cost for scale: ~0.8 ms per 600-char chunk (0.72–0.87 across thread arms, ~1,150–1,380 chunks/s), query encode 0.08 ms.
Same protocol as §05 — 800 lines, 32 seeds per cell, k=8, scattered decoys the hard case. The idf-weighted dense superposition holds rank under exactly the term-collision that inverts BM25's scoring.
| scattered decoys | system | P@1 | R@8 | mean rank | mean margin |
|---|---|---|---|---|---|
| 0 | leCore VSA | 1.00 | 1.00 | 1.00 | n/a |
| leCore BM25 | 1.00 | 1.00 | 1.00 | n/a | |
| 2 | leCore VSA | 0.81 | 1.00 | 1.22 | +0.059 |
| leCore BM25 | 0.19 | 1.00 | 2.09 | −0.014 | |
| 10 | leCore VSA | 0.09 | 1.00 | 3.22 | −0.273 |
| leCore BM25 | 0.09 | 0.91 | 5.69 | −0.348 | |
| 25 | leCore VSA | 0.03 | 0.69 | 7.41 | −0.460 |
| leCore BM25 | 0.03 | 0.22 | 11.97 | −0.444 |
The honest read of the win: at 10+ scattered decoys the VSA's precision@1 collapses too — 0.09 → 0.03. What it keeps is recall: at 25 scattered decoys the answer is still inside a k=8 context 69% of the time against BM25's 22%, before any of §05's k-raising. Clustered decoys cost it nothing at all: P@1 = 1.00 at 2, 10 and 25, where BM25 falls 1.00 → 0.72 → 0.25.
This is not "the VSA is good now." Its ranking generality is unchanged: SciFact nDCG@10 0.4160 against leCore BM25's 0.6705 and bge-base's 0.7404 — the same score that got the arm removed from the hybrid in §02. And at zero lexical overlap it is still not retrieval: J=0 recall@64 = 0.0964, reproduced here to the fourth decimal against §08's recorded run, position skew and all.
Any shared content word and it beats bge-base at deployed k (§08, J=.257: .958 at k=8) — on the cost axis nothing else touches (§01, §03).
Holds the needle under the term-collision that inverts BM25 — P@1 .81 vs .19 at two scattered decoys, 1.00 clustered throughout (this section).
The only system not flat at J=0 (.615 at k=8 → .846 at k=64, §08). Staged, not shipped — as an integrated stage it is unmeasured, and this page does not claim otherwise.
Method: local Apple M5 Pro (18 cores), Python 3.14.6 / numpy 2.5.0, $0 spent, wall clock 137 s. Load average was 5–6 throughout and is recorded per-section in results.json — a timing-contamination flag we publish rather than hide. Two thread arms, pinned single-thread and free, both reported; the ladder above uses the free arm (the single-thread floor was 3.7 ms, the single-thread wrapper 21.3 s — same shape). At 1k docs the auto index is exact and the wrapper is 2–4 ms; at 12k docs the forest kicks in and the wrapper is already 1.7–2.3 s. The rebuild dominates everywhere the forest exists.
§5's decoys were templated strings we wrote ourselves, which left one open question: does recall recovers with k survive corpora where the confusable passages are real? So: three BEIR corpora with real judged queries, leCore BM25 against GPU-embedded baselines in the same harness, hit@k swept from 1 to 200. The question is the crossover: at what k does leCore's hit rate reach what bge-base delivers at k=10? Answer: k = 32–64 on all three, at 3.2–6.4× the retrieved tokens.
leCore reaches .893 at k=64, clearing bge-base's .883 at k=10 — on the corpus where that took the most context. Real corpora land at gentler k than §5's worst-case templated decoys, where the required k was roughly the full count of competing passages.
| corpus | docs / queries | bge hit@10 | leCore hit@10 | crossover k | leCore hit@k | token cost |
|---|---|---|---|---|---|---|
| SciFact | 5,183 / 300 | 0.883 | 0.820 | k = 64 | 0.893 | 6.4× · ~17,857 tok/q |
| NFCorpus | 3,633 / 323 | 0.746 | 0.681 | k = 32 | 0.755 | 3.2× · ~9,725 tok/q |
| ArguAna | 8,674 / 1,401 | 0.884 | 0.685 | k = 48 | 0.894 | 4.8× · ~10,431 tok/q |
Do not read this as parity. Give bge-base the same k=64 and it scores .950 / .854 / .991 on the three corpora — more context helps everyone. What this bench establishes is that the recall-recovery lever survives real corpora, and what it costs: 3.2–6.4× tokens to match bge's k=10. The sellable number is cost per hit without a GPU or an embedding pass, not "we catch up."
Production now defaults to top_k=16 with a cap of 128 — every crossover k measured here (32–64) sits inside it. The old cap of 32 would have foreclosed SciFact's k=64 outright.
Method and the honest fine print: clean Vast RTX 4090, load average 1.5, embedding on GPU. bge-HNSW matched exact flat search on every corpus — identical on SciFact and ArguAna, within 0.007 hit on NFCorpus — so there is no ANN recall loss to exploit at these sizes. And on this box the embedders' index build was as fast or faster than our BM25 build (bge 11.3–15.5 s GPU vs leCore 15.8–50.5 s CPU): the §01 cost-to-first-answer win assumes no GPU is present. One boundary on everything above: the ranker is lexical — §08 measures the case where a question shares zero vocabulary with its answer, and the k lever does not survive it.
The loss, plainly: leCore's ranker is lexical. A question sharing zero vocabulary with its answer defeats it, and buying more context does not help. 64 needle/question pairs per overlap level, with the zero enforced in code, not by eyeballing — stemmed content-token sets must not intersect, no two content tokens may share a 4-character prefix, no shared numbers. Needles were embedded in ~222k tokens of real fineweb prose (~1.9k chunks), 3 positions × 2 seeds, through the deployed 600-char chunker. At measured Jaccard 0.000, leCore BM25 goes R@8 = .005 → R@64 = .023 — flat. Unlike §05 and §06, raising k recovers nothing, because there is no lexical anchor to rank on.
The measured boundary: the cliff is at exactly zero. One shared content word in four (the dashed line, measured J = .257) restores full recall at the deployed k — .96 at k=8, 1.00 at k=64 — ahead of bge-base at every k. And even the semantic reference struggles at true zero: bge-base reaches only .85 at k=64, not 1.0.
| measured overlap | system | R@8 | R@16 | R@32 | R@64 |
|---|---|---|---|---|---|
| J = 0.000 | leCore BM25 | 0.005 | 0.005 | 0.008 | 0.023 ← flat |
| leCore VSA | 0.008 | 0.023 | 0.052 | 0.096 ← argsort luck | |
| bge-base | 0.615 | 0.706 | 0.784 | 0.846 ← the reference struggles too | |
| J = 0.257 (~25%) | leCore BM25 | 0.958 | 0.987 | 0.995 | 1.000 ← ahead of bge at every k |
| bge-base | 0.881 | 0.913 | 0.958 | 0.968 | |
| J = 0.476 (~50%) | leCore BM25 | 1.000 | 1.000 | 1.000 | 1.000 |
| bge-base | 0.942 | 0.958 | 0.966 | 0.976 |
This bounds §05 and §07's recall-recovers-with-k story: the lever works when any lexical anchor exists, and does not exist at zero. Also honest: leCore's VSA arm's .10 at k=64 is position-skewed — .16 / .10 / .03 at start / middle / end — which is argsort luck, not retrieval. The changelog named this exact case as the likeliest failure before the bench was run; it is published here either way.
The failure needs exactly zero. At measured J = .257 leCore BM25 is .958 at k=8 → 1.000 at k=64, ahead of bge-base at every k, and at J = .476 everything saturates near 1.00. Real questions that share literally no content word with their answer are a corner of the space — but it is a corner we cannot serve, and now it is measured.
Method and the caveat to carry: pairs generated by Qwen2.5-7B-Instruct, then gate-enforced with rejects counted and published — at the zero level 65 rejected for stem overlap, 15 for shared prefixes, 8 for shared digits; 0 needles destroyed by chunking; 384 trials at J=0 and 378 at each other level. The no-shared-digit rule sometimes forces zero-overlap questions to alter quantities (a "50 new sensors" needle asked about as "30 extra devices"). That is fine for retrieval scoring — only chunk identity is scored — but the J=0 pairs are not quotable as QA pairs.
A tester in Moose's Telegram cloned leCore fresh, on his own hardware, and reproduced our BM25 numbers to four decimals. He also ran the column we never had: leCore's hand-rolled BM25 against the one people actually pip install. We re-ran all of it here — third machine — and the answer is not the one the branding implies. It is not the hologram. It is a stoplist and a stemmer.
| nDCG@10 | SciFact | NFCorpus | ArguAna |
|---|---|---|---|
| leCore BM25 0.2.0 | 0.6689 | 0.3179 | 0.4867 |
| leCore BM25 0.1.0 — query terms deduped | 0.6679 | 0.3185 | 0.4300 |
pip rank_bm25, as its README uses it |
0.5597 | 0.2671 | 0.3448 |
pip rank_bm25 + leCore's tokenizer |
0.6664 | 0.3192 | 0.4835 |
pip bm25s, its documented defaults |
0.6617 | 0.3064 | 0.4839 |
pip bm25s + leCore's tokenizer |
0.6689 | 0.3175 | 0.4862 |
the expand knob, delta |
+0.0026 | −0.0014 | +0.0008 |
+10.9, +5.1, +14.2 points. rank_bm25 ships no tokenizer at all, so its README demonstrates doc.lower().split() — and that is what most people deploy. The number is real. It is also a comparison against the weakest baseline available, which is why the tester's fairer pip config only gave us +1.6/+1.2/+4.4. Read row 5 next.
bm25s is maintained, and its documented defaults already lowercase and drop English stopwords. Margin: +0.7, +1.2, +0.3. On ArguAna it beat our 0.1.0 outright — 0.4839 to 0.4300 — and it took the qtf fix below just to pull level. The tokenizer is the entire advantage, and against a library that already tokenizes, that advantage is worth about a point. If you already run bm25s, there is no numerical reason to switch.
The tester got +1.6/+1.2/+4.4. We published +10.9/+5.1/+14.2. Neither of us was wrong — we picked different baselines, and the phrase "the standard pip library" turns out to cover at least four things. Same leCore, same tasks, same scorer:
| what you call "pip BM25" | SciFact | NFCorpus | ArguAna |
|---|---|---|---|
rank_bm25, README tokenization |
0.5597 | 0.2671 | 0.3461 |
| scikit-learn TF-IDF, defaults | 0.5780 | 0.2903 | 0.4269 |
Whoosh StandardAnalyzer + BM25F |
0.6504 | 0.3054 | not run |
bm25s, documented defaults |
0.6617 | 0.3064 | 0.4839 |
| leCore BM25 0.2.0 | 0.6689 | 0.3179 | 0.4867 |
The advantage swings from +14.2 points to +0.3 on nothing but which library you call standard. Back out the tester's implied baseline — 0.6519 / 0.3047 / 0.3865 — and it sits in the sane-analyzer band, near Whoosh and bm25s. That is the whole discrepancy. Our headline was a statement about rank_bm25's README, not about BM25.
One comparison we deliberately do not make: the published BEIR Elasticsearch BM25 baselines (SciFact 0.620, NFCorpus 0.297, ArguAna 0.441). We beat all three on paper — and it would be dishonest to say so, because those are full-corpus and every number on this page uses a 1000-document scoring pool. That is an easier problem. Full-corpus runs are queued.
This section used to end in a defeat, and it stayed up: row 3 did not just match leCore on ArguAna, it won 0.4835 vs 0.4300. The cause was one line, and it was ours:
0.1.0: for t in sorted(set(q_terms)): # query terms DEDUPED
rank_bm25: for q in query: # every occurrence counts
0.2.0: for t, c in sorted(Counter(q_terms).items()):
out[idxs] += float(c) * wts # q occurrences -> q x weightleCore was discarding query-term frequency — the query side of BM25. On keyword-length queries that is invisible, because terms almost never repeat. On ArguAna, where every "query" is a full argument passage, it was throwing away real signal. Counting terms instead of deduping recovers +0.0566 and puts leCore past pip-with-our-tokenizer, 0.4867 vs 0.4835.
| task | query toks | repeat rate | delta |
|---|---|---|---|
| NFCorpus | 2.7 | 0.003 | −0.0006 |
| Touche2020 | 4.2 | 0.006 | +0.0000 |
| TREC-COVID | 7.9 | 0.008 | −0.0002 |
| SCIDOCS | 7.9 | 0.016 | −0.0018 |
| FiQA | 7.3 | 0.019 | −0.0009 |
| SciFact | 9.5 | 0.028 | +0.0011 |
| ArguAna | 121.6 | 0.230 | +0.0566 |
Do not over-read the correlation. Against mean query length it is r = 0.998, which looks like a law — but it is carried by one point. Drop ArguAna and it collapses to r = 0.155. All we can honestly claim is a bound: harmless up to ~9.5 tokens / 0.028 repeat rate, worth 5.7 points at 121.6 / 0.230. Nothing we ran says where in between it starts to bite. Shipped as lecore-bm25 0.2.0; harness qtf_law_bench.py, deterministic across three PYTHONHASHSEEDs, and the package's own scorer reproduces the independent reimplementation to six decimals.
Method: BEIR via the mteb/* HF datasets, scored with pytrec_eval ndcg_cut.10 — the scorer mteb uses underneath — 1000-doc pool, ignore_identical_ids on ArguAna, same title+body doc text as the recorded runs. Our previously published 0.6705 turns out to be the expand=True variant; plain is 0.6679. Harness: bm25_vs_pip_bench.py and qtf_law_bench.py. The tokenizer is now installable on its own — pip install lecore-bm25, MIT, credit to AnOversizedMooseWithSocks, whose code it is.
Every k-curve above — §07, §08 — measures the same task: one needle, one haystack, is it in the top k. On that task raising k genuinely rescues recall, and we leaned on it. So we asked what happens when memory is asked to do the other things memory does: notice a fact was replaced, gather a fact that is scattered, and follow one that points somewhere else. 800 lines, 200 docs, 32 trials per cell, k from 8 to 128, leCore against a substring-matching floor.
Live fact ranked first, at every k from 8 to 128 and every revision count from 1 to 8. Not "degrades" — flat zero. The live fact is already inside k (1.000); it is simply ranked below the dead ones. More k retrieves more corpses.
Recall of an n-item set is min(1, k/n), dead on: 8 of 16 = .500, 8 of 32 = .250, 8 of 64 = .125. Nothing is mis-ranked, the window is just too small. Set k above the fan-out and it is perfect. This is the lever working as advertised.
Both hops in one shot, k=8 → k=128. Sixteen times the context to still miss a quarter of them. Query twice instead — hop one, then hop two — and it is 1.000 at every k, including 8. The lever was "ask again", not "look wider".
The vault code is set in 2019 and changed four times. Ask "what is the current vault access code?" and you get the 2019 one, 100% of the time. Two things cause it, and neither is fixable with k:
The word "current" matches nothing. No document contains it. There is no recency term to score, so the query's one temporal word is dead weight.
The dead fact is phrased more tersely. The original reads "the vault access code is X"; each revision reads "has been changed to Y". Those extra tokens lengthen the document, and BM25's length normalisation penalises exactly that. The superseded version wins on brevity. Phrase the revisions identically and all versions tie to the bit, and insertion order decides instead — which is oldest-first. Neither route encodes recency.
Which is why the honest headline is not "leCore fails": the substring floor fails identically, 0.000 across the board. BM25 scores relevance. Supersession is a question about time. Any pure-relevance ranker — ours, pip's, or a vector index — answers it wrong unless something outside the ranker carries the timestamp. Our determinism guarantee makes it worse in one specific way: a coin-flipping ranker would get it right sometimes, and we get it wrong reproducibly.
The k-curves in §07 and §08 stand — they were measured honestly, and on the task they measure, k is the right dial. What was never true is the generalisation we let it imply. Read them as: "if the fact is in there once, raise k and you will find it." Not as a claim about memory.
Fixes we have not shipped and are not claiming: a recency prior at scoring time, or key-level dedupe at write time so a superseded fact stops being retrievable at all. Both are outside BM25. The multi-hop result at least ships today — two queries, any k, 1.000. Harness: memory_consistency_bench.py, 32 trials per cell, both rankers, seeds recorded.
Three people told us we were wrong about something. Read each against the actual claim: much cheaper, and usable where the alternative cannot run at all. A confirmed counterclaim is only a defeat if it contradicts something we asserted — several of these are true and were priced in from the start.
| claim | who | verdict | does it hit the pitch? |
|---|---|---|---|
| BM25 is not SOTA — it's ~free and runs on a Raspberry Pi | Shaw | CONFIRMED — bge-base wins 4/4 | NO — quality parity was never claimed. This is the trade. |
| Compare against HNSW, not flat FAISS | Shaw | CONFIRMED — HNSW is sub-linear as stated | NO — HNSW fixes query time, not the O(corpus) embed we skip. |
| Embeddings = quality; ANN = performance | Shaw | CONFIRMED — exactly the split found | NO — we compete on the third axis, cost. |
| leCore recall trails FAISS | Cotten | CONFIRMED — our dense arm scored 0.416 | NO — and we deleted that arm. |
| Test adversarially, with near-matches | Cotten | CONFIRMED at k=8 — P@1 1.00 → 0.19 at two decoys | YES, briefly — fixed by raising k. R@24 = 1.00 at 25 decoys; holds on real BEIR corpora at crossover k ≤ 64 (§07). |
| You are ignoring leCore | Moose | CONFIRMED twice — chunker and encoder hand-rolled | Process, not product — both now use leCore. |
| 518 ms per query | Cotten | EXPLAINED — §06: prebuilt index 2–16 ms, deployed wrapper 20.5 s. His number was charitable | Deployment, not algorithm — the wrapper rebuilt the forest per query. Fix (a cache) shipping; 20.5 s stands until re-measured. |
| Never skip the VSA — profile it over 1024d; the _Index wrapper isn't optimized | Cotten | CONFIRMED on the wrapper — ~4,200× the math floor (§06) | NO — and the profile he asked for handed the VSA its first outright win: decoy resistance (§06). |
| Embeddings win everywhere | implied | REFUTED on cost — 13.32 s vs 182.90 s TTFA | This is the pitch. |
13.7× faster to first answer at 1M tokens, 98× fewer prompt tokens, with enough headroom to raise top_k 8× and still be 12.3× cheaper than sending the corpus.
24/24 answered at 1M–5M where the control returns UNRUNNABLE at every tier.
The adversarial bench is the one that genuinely landed, and only at the shipped top_k=8: a config bug, not an architecture defeat. The real-corpora shotgun (§07) then confirmed the recovery lever on BEIR data at k ≤ 64 — with the caveat that corpus-size scaling beyond ~8.7k docs is still unmeasured. The semantic NIAH (§08) then drew the lever's hard boundary: it requires at least one shared content word, and at measured zero overlap raising k recovers nothing — while one word in four restores full recall at the deployed k, ahead of bge-base. That boundary was named in the changelog as the likeliest failure before it was run, and it is published either way.
§07 retired the templated-decoy caveat: on real BEIR corpora, recall recovers within k ≤ 64. But those corpora top out at 8,674 documents. Whether crossover k grows with corpus size — 100k docs, millions — is unmeasured, and it is still the thing that could break the cost/recall tradeoff.
§07 prices a single query: matching bge's hit@10 costs ~9.7k–17.9k retrieved tokens against corpora of ~0.8M–1.9M tokens — far cheaper than sending the corpus. What is unmeasured is amortisation: leCore pays 3.2–6.4× tokens on every query, the embedder pays its index once. At what query volume does bge become the cheaper system end to end?
The LM NIAH has now run (§04's follow-up panel) and HRR never localized at any length — but the pair is not loss-matched, so operator vs plain LM quality is confounded. A matched-loss rerun is in flight.
The latency reconciliation itself is done — §06 decomposed the 100× disagreement into a per-query forest rebuild in the deployed wrapper. The fix, caching the built forest across queries, is shipping but not verified deployed; the honest deployed number remains 20.5 s at 100k docs until the cached path is measured live.
n=9 per cell at 60k–500k, n=6 past it, single runs at temperature 0. Cost benchmarks run on an unloaded box — an earlier set was discarded after load average 67 inflated FAISS index times by 3×. Only the harness is parallelised: BM25 fits single-threaded because that is how leCore ships it, and the embedder gets its own batching because that is how it ships. Model: gemini-2.5-flash via OpenRouter. elizaOS's own harness could not be run — bun install fails on three private @elizaos/* packages — so the numbers are ours, our runner, their methodology.