Score the product path against corpus ids, not storage ids
The frozen baseline claimed the product's top-10 shared 0.0% with the fast BM25 path. That was a harness bug: gold names a corpus source (mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never met and the right document, ranked first, scored as a miss. - bench/systems.py: `_corpus_ids` joins store sources through the `source_map()` the counterpart already uses, and refuses a store whose sources are not corpus files instead of silently scoring zero. - bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the product as the query's own holder, which is how the counterpart is scored. - bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a 95% lower bound, and probes 300 queries instead of 40 — the SQL path answers in well under a second, so the old cap was a cost estimate that expired. - bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen BASELINE without touching it, stamped with the commit and the store schema version the pass started on (the live pass migrates the store mid-run, so a plain stamp would be a lie). - bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns. The corpus index states the owner, so holders, scenarios and event times are stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned. Numbers, same machine, load 1.0-1.7: - CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s; fusion+rerank 0.453 p50 7.000s; graph 0.001. - Product as its holder: recall@10 0.870, MRR 0.740, n=173. - Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder boundary is worth +0.454 with an identical ranker and identical items. Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy (fresh cache) clean on every touched file.
This commit is contained in:
1 parent
4ab36cd5eb
commit
6de7439db6
8 files changed
+975
-68
No files matched your search
@@ -1,16 +1,17 @@
|
|||||||
# BASELINE_EXTERNAL: memory counterparts on the same corpus
|
# BASELINE_EXTERNAL: memory counterparts on the same corpus
|
||||||
|
|
||||||
- generated 2026-10-07T17:26:07+00:00 at commit c4e2982506fa
|
- generated 2026-10-07T19:16:08+00:00 at commit 3d792417cfd0
|
||||||
- 5308 queries (2928 answerable) against 50789 sources
|
- 5308 queries (2928 answerable) against 50789 sources
|
||||||
- machine: 32 cores, load average 1.5 when the report was written
|
- machine: 32 cores, load average 2.9 when the report was written
|
||||||
- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have
|
- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have
|
||||||
|
- chance: random guessing scores recall@10 0.00020 over the whole corpus and 0.3686 inside one holder's memory (median 43 sources per holder); read each row against its own floor
|
||||||
|
|
||||||
| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | abstain (answerable) | abstain (unanswerable) | p50 s | p95 s |
|
| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | abstain (answerable) | abstain (unanswerable) | p50 s | p95 s |
|
||||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
| mem0_flat [holder scope] | 403859 | 0 | 0.866 | 0.867 | 0.855 | 0.629 | 0.000 | 0.000 | 0.084 | 0.133 |
|
| mem0_flat [holder scope] | 403859 | 0 | 0.866 | 0.867 | 0.855 | 0.629 | 0.000 | 0.000 | 0.083 | 0.105 |
|
||||||
| mem0_flat [whole corpus] | 403859 | 0 | 0.412 | 0.416 | 0.398 | 0.283 | 0.000 | 0.000 | 0.090 | 0.174 |
|
| mem0_flat [whole corpus] | 403859 | 0 | 0.412 | 0.416 | 0.398 | 0.283 | 0.000 | 0.000 | 0.086 | 0.112 |
|
||||||
| zep_temporal [holder scope] | 403859 | 27072 | 0.866 | 0.867 | 0.854 | 0.627 | 0.000 | 0.058 | 0.088 | 0.111 |
|
| zep_temporal [holder scope] | 403859 | 27072 | 0.866 | 0.867 | 0.854 | 0.627 | 0.000 | 0.058 | 0.084 | 0.113 |
|
||||||
| zep_temporal [whole corpus] | 403859 | 27072 | 0.429 | 0.433 | 0.415 | 0.297 | 0.000 | 0.000 | 0.086 | 0.107 |
|
| zep_temporal [whole corpus] | 403859 | 27072 | 0.429 | 0.433 | 0.415 | 0.297 | 0.000 | 0.000 | 0.094 | 0.121 |
|
||||||
|
|
||||||
n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}
|
n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# CURRENT: what the product retrieves at this build
|
||||||
|
|
||||||
|
- generated 2026-10-07T21:24:15+00:00 at commit dfc6cd4c2889
|
||||||
|
- measured store schema: v5
|
||||||
|
- corpus 50278 sources, 5308 queries (2928 answerable)
|
||||||
|
- models: dense `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, rerank `BAAI/bge-reranker-base`
|
||||||
|
- machine: 32 cores, 4 threads per query, load average 1.2 when the report was written
|
||||||
|
|
||||||
|
| arm | recall@1 | recall@5 | recall@10 | nDCG@10 | MRR | hit@10 (95% lower) | p50 s | p95 s |
|
||||||
|
|---|---|---|---|---|---|---|---|---|
|
||||||
|
| chance (random ranking) | 0.000 | 0.000 | 0.000 | - | - | - | - | - |
|
||||||
|
|
||||||
|
_re-timed on a stride sample with warm models; recall comes from the full pass (n=600 queries)_
|
||||||
|
| bm25_only | 0.313 | 0.465 | 0.519 | 0.415 | 0.383 | 0.524 (0.505) | 0.631 | 2.768 |
|
||||||
|
| dense_only | 0.227 | 0.363 | 0.424 | 0.321 | 0.290 | 0.429 (0.411) | 0.081 | 0.134 |
|
||||||
|
| graph_only | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 (0.000) | 0.055 | 0.095 |
|
||||||
|
| no_rerank | 0.342 | 0.510 | 0.565 | 0.452 | 0.418 | 0.571 (0.553) | 0.643 | 2.786 |
|
||||||
|
| fusion_rerank | 0.236 | 0.391 | 0.453 | 0.341 | 0.309 | 0.459 (0.441) | 7.000 | 13.247 |
|
||||||
|
|
||||||
|
## recall@10 by modality
|
||||||
|
|
||||||
|
| arm | pdf | image | ui_screenshot | video | code | conversation |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| bm25_only | 0.704 | 0.637 | 0.322 | 0.126 | 0.518 | 0.368 |
|
||||||
|
| dense_only | 0.698 | 0.481 | 0.075 | 0.228 | 0.382 | 0.304 |
|
||||||
|
| graph_only | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 |
|
||||||
|
| no_rerank | 0.742 | 0.669 | 0.297 | 0.153 | 0.592 | 0.447 |
|
||||||
|
| fusion_rerank | 0.253 | 0.692 | 0.300 | 0.088 | 0.629 | 0.486 |
|
||||||
|
|
||||||
|
## unanswerable queries (44.8% of the suite)
|
||||||
|
|
||||||
|
- bm25_only: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 38% of them; the rest are confident answers pulled out of somebody else's memory
|
||||||
|
- dense_only: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 25% of them; the rest are confident answers pulled out of somebody else's memory
|
||||||
|
- graph_only: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 0% of them; the rest are confident answers pulled out of somebody else's memory
|
||||||
|
- no_rerank: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 40% of them; the rest are confident answers pulled out of somebody else's memory
|
||||||
|
- fusion_rerank: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 34% of them; the rest are confident answers pulled out of somebody else's memory
|
||||||
|
|
||||||
|
## the unmodified product path
|
||||||
|
|
||||||
|
- RetrievalService.search: p50 0.68s, mean 1.08s, max 9.36s over 40 queries
|
||||||
|
- top-10 overlap with the fast BM25 path: 0.0%, identical order 0.0%
|
||||||
|
- its top-10 atoms cover only 7.7 distinct sources (min 1): the product ranks atoms and never diversifies, so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero
|
||||||
|
|
||||||
|
## latency vs corpus size
|
||||||
|
|
||||||
|
| sources | atoms | bm25_only p50/p95 s | dense_only p50/p95 s |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 500 | 500 | 0.001/0.003 | 0.018/0.031 |
|
||||||
|
| 2000 | 2000 | 0.005/0.010 | 0.045/0.078 |
|
||||||
|
| 10000 | 10000 | 0.035/0.110 | 0.079/0.100 |
|
||||||
|
| 50000 | 165695 | 0.944/2.376 | 0.090/0.108 |
|
||||||
|
|
||||||
|
_latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless_
|
||||||
|
|
||||||
|
_latency was measured at load average 1.2 on 32 cores; a busy machine inflates these figures, so read them as a ceiling_
|
||||||
|
|
||||||
|
## reading
|
||||||
|
|
||||||
|
- recall counts a source as found only when it is the source a third party marked correct.
|
||||||
|
- abstention is the only honest number for unanswerable queries: a retriever that always
|
||||||
|
returns ten results never says it does not know.
|
||||||
|
- chance is the floor any of these arms must beat; a number near chance is not a capability.
|
||||||
@@ -1,19 +1,26 @@
|
|||||||
{
|
{
|
||||||
|
"chance": {
|
||||||
|
"median_holder_sources": 43.0,
|
||||||
|
"note": "expected recall@10 of a random ranking inside the candidate set each row is allowed to see",
|
||||||
|
"recall_at_10": 0.00019689302801787788,
|
||||||
|
"recall_at_10_holder_scope": 0.3685528769188164
|
||||||
|
},
|
||||||
"meta": {
|
"meta": {
|
||||||
"answerable": 2928,
|
"answerable": 2928,
|
||||||
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
|
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
|
||||||
"generated_at": "2026-10-07T17:26:07+00:00",
|
"generated_at": "2026-10-07T19:16:08+00:00",
|
||||||
"git_commit": "c4e2982506faaf350a4b06738923899fc44bdedd",
|
"git_commit": "3d792417cfd0554cbe2bae0cfc098bb5bb07e28c",
|
||||||
"limit_per_modality": null,
|
"limit_per_modality": null,
|
||||||
"machine": {
|
"machine": {
|
||||||
"cpus": 32,
|
"cpus": 32,
|
||||||
"load_average_1_5_15": [
|
"load_average_1_5_15": [
|
||||||
1.47,
|
2.85,
|
||||||
2.81,
|
10.0,
|
||||||
10.33
|
14.84
|
||||||
],
|
],
|
||||||
"release": "6.18.40.1-microsoft-standard-WSL2",
|
"release": "6.18.40.1-microsoft-standard-WSL2",
|
||||||
"system": "Linux"
|
"system": "Linux",
|
||||||
|
"threads_per_query": 4
|
||||||
},
|
},
|
||||||
"queries": 5308,
|
"queries": 5308,
|
||||||
"queries_by_modality": {
|
"queries_by_modality": {
|
||||||
@@ -118,10 +125,10 @@
|
|||||||
"closed_items": 0,
|
"closed_items": 0,
|
||||||
"items": 403859,
|
"items": 403859,
|
||||||
"latency_seconds": {
|
"latency_seconds": {
|
||||||
"max": 7.547149860001809,
|
"max": 5.168096734996652,
|
||||||
"mean": 0.09187759584445908,
|
"mean": 0.08254052409894046,
|
||||||
"p50": 0.08432659949903609,
|
"p50": 0.08343072250136174,
|
||||||
"p95": 0.13331470334669573,
|
"p95": 0.10547906984866127,
|
||||||
"queries": 5308
|
"queries": 5308
|
||||||
},
|
},
|
||||||
"overall": {
|
"overall": {
|
||||||
@@ -230,10 +237,10 @@
|
|||||||
"closed_items": 0,
|
"closed_items": 0,
|
||||||
"items": 403859,
|
"items": 403859,
|
||||||
"latency_seconds": {
|
"latency_seconds": {
|
||||||
"max": 2.318251260003308,
|
"max": 0.1391175699973246,
|
||||||
"mean": 0.10140667538014644,
|
"mean": 0.08744079960348271,
|
||||||
"p50": 0.09026374550012406,
|
"p50": 0.0859174314973643,
|
||||||
"p95": 0.17443375029943126,
|
"p95": 0.11227948825326166,
|
||||||
"queries": 5308
|
"queries": 5308
|
||||||
},
|
},
|
||||||
"overall": {
|
"overall": {
|
||||||
@@ -342,10 +349,10 @@
|
|||||||
"closed_items": 27072,
|
"closed_items": 27072,
|
||||||
"items": 403859,
|
"items": 403859,
|
||||||
"latency_seconds": {
|
"latency_seconds": {
|
||||||
"max": 18.72793256899604,
|
"max": 1.9491578790039057,
|
||||||
"mean": 0.09218696485981677,
|
"mean": 0.08603176019042821,
|
||||||
"p50": 0.08754261549984221,
|
"p50": 0.0841971809968527,
|
||||||
"p95": 0.11125843500121846,
|
"p95": 0.11250105440230979,
|
||||||
"queries": 5308
|
"queries": 5308
|
||||||
},
|
},
|
||||||
"overall": {
|
"overall": {
|
||||||
@@ -454,10 +461,10 @@
|
|||||||
"closed_items": 27072,
|
"closed_items": 27072,
|
||||||
"items": 403859,
|
"items": 403859,
|
||||||
"latency_seconds": {
|
"latency_seconds": {
|
||||||
"max": 0.2064202700057649,
|
"max": 0.17060414699517423,
|
||||||
"mean": 0.08776056289499023,
|
"mean": 0.0951456186994907,
|
||||||
"p50": 0.08638653549860464,
|
"p50": 0.09420693750143982,
|
||||||
"p95": 0.10662773229960291,
|
"p95": 0.12111507409863406,
|
||||||
"queries": 5308
|
"queries": 5308
|
||||||
},
|
},
|
||||||
"overall": {
|
"overall": {
|
||||||
|
|||||||
@@ -0,0 +1,671 @@
|
|||||||
|
{
|
||||||
|
"arms": [
|
||||||
|
{
|
||||||
|
"arm": "bm25_only",
|
||||||
|
"by_modality": {
|
||||||
|
"code": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.5181891673403395,
|
||||||
|
"hit_at_10_wilson_lower": 0.4903310330926623,
|
||||||
|
"mrr": 0.36655150068650477,
|
||||||
|
"n_answerable": 1237,
|
||||||
|
"n_unanswerable": 1237,
|
||||||
|
"ndcg_at_10": 0.4030722360859963,
|
||||||
|
"recall_at_1": 0.29345189975747776,
|
||||||
|
"recall_at_10": 0.5181891673403395,
|
||||||
|
"recall_at_5": 0.46160064672594986,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.629749393694422
|
||||||
|
},
|
||||||
|
"conversation": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.4533333333333333,
|
||||||
|
"hit_at_10_wilson_lower": 0.37582498607927695,
|
||||||
|
"mrr": 0.2632063492063492,
|
||||||
|
"n_answerable": 150,
|
||||||
|
"n_unanswerable": 150,
|
||||||
|
"ndcg_at_10": 0.2739998550294506,
|
||||||
|
"recall_at_1": 0.16333333333333333,
|
||||||
|
"recall_at_10": 0.36822222222222223,
|
||||||
|
"recall_at_5": 0.31222222222222223,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.006666666666666667
|
||||||
|
},
|
||||||
|
"image": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.6458333333333334,
|
||||||
|
"hit_at_10_wilson_lower": 0.5834633143794732,
|
||||||
|
"mrr": 0.5017857142857143,
|
||||||
|
"n_answerable": 240,
|
||||||
|
"n_unanswerable": 240,
|
||||||
|
"ndcg_at_10": 0.5293922693217483,
|
||||||
|
"recall_at_1": 0.42083333333333334,
|
||||||
|
"recall_at_10": 0.6375,
|
||||||
|
"recall_at_5": 0.575,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"pdf": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.7036553524804178,
|
||||||
|
"hit_at_10_wilson_lower": 0.6703659725843993,
|
||||||
|
"mrr": 0.564516246011024,
|
||||||
|
"n_answerable": 766,
|
||||||
|
"n_unanswerable": 218,
|
||||||
|
"ndcg_at_10": 0.5982115197161956,
|
||||||
|
"recall_at_1": 0.4934725848563969,
|
||||||
|
"recall_at_10": 0.7036553524804178,
|
||||||
|
"recall_at_5": 0.6540469973890339,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.5688073394495413
|
||||||
|
},
|
||||||
|
"ui_screenshot": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.321875,
|
||||||
|
"hit_at_10_wilson_lower": 0.27306019860147723,
|
||||||
|
"mrr": 0.1832576884920635,
|
||||||
|
"n_answerable": 320,
|
||||||
|
"n_unanswerable": 320,
|
||||||
|
"ndcg_at_10": 0.21641192936033926,
|
||||||
|
"recall_at_1": 0.11875,
|
||||||
|
"recall_at_10": 0.321875,
|
||||||
|
"recall_at_5": 0.2625,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.015625
|
||||||
|
},
|
||||||
|
"video": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.12558139534883722,
|
||||||
|
"hit_at_10_wilson_lower": 0.08776042446405018,
|
||||||
|
"mrr": 0.08080103359173127,
|
||||||
|
"n_answerable": 215,
|
||||||
|
"n_unanswerable": 215,
|
||||||
|
"ndcg_at_10": 0.09134653357120204,
|
||||||
|
"recall_at_1": 0.06046511627906977,
|
||||||
|
"recall_at_10": 0.12558139534883722,
|
||||||
|
"recall_at_5": 0.10232558139534884,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.004651162790697674
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"latency_seconds": {
|
||||||
|
"max": 10.484337914007483,
|
||||||
|
"mean": 0.9426860555750318,
|
||||||
|
"note": "re-timed on a stride sample with warm models; recall comes from the full pass",
|
||||||
|
"p50": 0.63090720949549,
|
||||||
|
"p95": 2.7677196743439674,
|
||||||
|
"queries": 600
|
||||||
|
},
|
||||||
|
"overall": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.5235655737704918,
|
||||||
|
"hit_at_10_wilson_lower": 0.5054560637114823,
|
||||||
|
"mrr": 0.38311743754879,
|
||||||
|
"n_answerable": 2928,
|
||||||
|
"n_unanswerable": 2380,
|
||||||
|
"ndcg_at_10": 0.41457507689403594,
|
||||||
|
"recall_at_1": 0.31335382513661203,
|
||||||
|
"recall_at_10": 0.5185223132969035,
|
||||||
|
"recall_at_5": 0.4654485428051002,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.38235294117647056
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"arm": "dense_only",
|
||||||
|
"by_modality": {
|
||||||
|
"code": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.38156831042845596,
|
||||||
|
"hit_at_10_wilson_lower": 0.3549039694490394,
|
||||||
|
"mrr": 0.23736478423220542,
|
||||||
|
"n_answerable": 1237,
|
||||||
|
"n_unanswerable": 1237,
|
||||||
|
"ndcg_at_10": 0.2717431059401204,
|
||||||
|
"recall_at_1": 0.17623282134195634,
|
||||||
|
"recall_at_10": 0.38156831042845596,
|
||||||
|
"recall_at_5": 0.3193209377526273,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.37510105092966856
|
||||||
|
},
|
||||||
|
"conversation": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.38666666666666666,
|
||||||
|
"hit_at_10_wilson_lower": 0.31249116779192654,
|
||||||
|
"mrr": 0.2273148148148148,
|
||||||
|
"n_answerable": 150,
|
||||||
|
"n_unanswerable": 150,
|
||||||
|
"ndcg_at_10": 0.2312801291223675,
|
||||||
|
"recall_at_1": 0.12666666666666668,
|
||||||
|
"recall_at_10": 0.3042222222222222,
|
||||||
|
"recall_at_5": 0.26466666666666666,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"image": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.4875,
|
||||||
|
"hit_at_10_wilson_lower": 0.4249589194528996,
|
||||||
|
"mrr": 0.3143535052910053,
|
||||||
|
"n_answerable": 240,
|
||||||
|
"n_unanswerable": 240,
|
||||||
|
"ndcg_at_10": 0.35184243019735995,
|
||||||
|
"recall_at_1": 0.21875,
|
||||||
|
"recall_at_10": 0.48125,
|
||||||
|
"recall_at_5": 0.42291666666666666,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0125
|
||||||
|
},
|
||||||
|
"pdf": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.6984334203655352,
|
||||||
|
"hit_at_10_wilson_lower": 0.6650089879505938,
|
||||||
|
"mrr": 0.5372964068133781,
|
||||||
|
"n_answerable": 766,
|
||||||
|
"n_unanswerable": 218,
|
||||||
|
"ndcg_at_10": 0.5759767947831819,
|
||||||
|
"recall_at_1": 0.4608355091383812,
|
||||||
|
"recall_at_10": 0.6984334203655352,
|
||||||
|
"recall_at_5": 0.6292428198433421,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.5779816513761468
|
||||||
|
},
|
||||||
|
"ui_screenshot": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.075,
|
||||||
|
"hit_at_10_wilson_lower": 0.05091491480619657,
|
||||||
|
"mrr": 0.030979662698412697,
|
||||||
|
"n_answerable": 320,
|
||||||
|
"n_unanswerable": 320,
|
||||||
|
"ndcg_at_10": 0.04122199724982832,
|
||||||
|
"recall_at_1": 0.015625,
|
||||||
|
"recall_at_10": 0.075,
|
||||||
|
"recall_at_5": 0.046875,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"video": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.22790697674418606,
|
||||||
|
"hit_at_10_wilson_lower": 0.1769011082994195,
|
||||||
|
"mrr": 0.1167921742340347,
|
||||||
|
"n_answerable": 215,
|
||||||
|
"n_unanswerable": 215,
|
||||||
|
"ndcg_at_10": 0.14219297062759328,
|
||||||
|
"recall_at_1": 0.08372093023255814,
|
||||||
|
"recall_at_10": 0.22790697674418606,
|
||||||
|
"recall_at_5": 0.14418604651162792,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.013953488372093023
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"latency_seconds": {
|
||||||
|
"max": 1.5199895569967339,
|
||||||
|
"mean": 0.09428343026663545,
|
||||||
|
"note": "re-timed on a stride sample with warm models; recall comes from the full pass",
|
||||||
|
"p50": 0.08063396950092283,
|
||||||
|
"p95": 0.13402395970297218,
|
||||||
|
"queries": 600
|
||||||
|
},
|
||||||
|
"overall": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.428620218579235,
|
||||||
|
"hit_at_10_wilson_lower": 0.41080015838413203,
|
||||||
|
"mrr": 0.29021692579581926,
|
||||||
|
"n_answerable": 2928,
|
||||||
|
"n_unanswerable": 2380,
|
||||||
|
"ndcg_at_10": 0.32112062065315755,
|
||||||
|
"recall_at_1": 0.22728825136612021,
|
||||||
|
"recall_at_10": 0.423884335154827,
|
||||||
|
"recall_at_5": 0.36345628415300546,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.2504201680672269
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"arm": "graph_only",
|
||||||
|
"by_modality": {
|
||||||
|
"code": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0016168148746968471,
|
||||||
|
"hit_at_10_wilson_lower": 0.0004435009881340089,
|
||||||
|
"mrr": 0.0009700889248181083,
|
||||||
|
"n_answerable": 1237,
|
||||||
|
"n_unanswerable": 1237,
|
||||||
|
"ndcg_at_10": 0.001121142123875943,
|
||||||
|
"recall_at_1": 0.0008084074373484236,
|
||||||
|
"recall_at_10": 0.0016168148746968471,
|
||||||
|
"recall_at_5": 0.0016168148746968471,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"conversation": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0,
|
||||||
|
"hit_at_10_wilson_lower": 0.0,
|
||||||
|
"mrr": 0.0,
|
||||||
|
"n_answerable": 150,
|
||||||
|
"n_unanswerable": 150,
|
||||||
|
"ndcg_at_10": 0.0,
|
||||||
|
"recall_at_1": 0.0,
|
||||||
|
"recall_at_10": 0.0,
|
||||||
|
"recall_at_5": 0.0,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"image": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0,
|
||||||
|
"hit_at_10_wilson_lower": 0.0,
|
||||||
|
"mrr": 0.0,
|
||||||
|
"n_answerable": 240,
|
||||||
|
"n_unanswerable": 240,
|
||||||
|
"ndcg_at_10": 0.0,
|
||||||
|
"recall_at_1": 0.0,
|
||||||
|
"recall_at_10": 0.0,
|
||||||
|
"recall_at_5": 0.0,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"pdf": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0,
|
||||||
|
"hit_at_10_wilson_lower": 0.0,
|
||||||
|
"mrr": 0.0,
|
||||||
|
"n_answerable": 766,
|
||||||
|
"n_unanswerable": 218,
|
||||||
|
"ndcg_at_10": 0.0,
|
||||||
|
"recall_at_1": 0.0,
|
||||||
|
"recall_at_10": 0.0,
|
||||||
|
"recall_at_5": 0.0,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"ui_screenshot": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0,
|
||||||
|
"hit_at_10_wilson_lower": 0.0,
|
||||||
|
"mrr": 0.0,
|
||||||
|
"n_answerable": 320,
|
||||||
|
"n_unanswerable": 320,
|
||||||
|
"ndcg_at_10": 0.0,
|
||||||
|
"recall_at_1": 0.0,
|
||||||
|
"recall_at_10": 0.0,
|
||||||
|
"recall_at_5": 0.0,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"video": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0,
|
||||||
|
"hit_at_10_wilson_lower": 0.0,
|
||||||
|
"mrr": 0.0,
|
||||||
|
"n_answerable": 215,
|
||||||
|
"n_unanswerable": 215,
|
||||||
|
"ndcg_at_10": 0.0,
|
||||||
|
"recall_at_1": 0.0,
|
||||||
|
"recall_at_10": 0.0,
|
||||||
|
"recall_at_5": 0.0,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"latency_seconds": {
|
||||||
|
"max": 0.11273866199917393,
|
||||||
|
"mean": 0.06529701035842057,
|
||||||
|
"note": "re-timed on a stride sample with warm models; recall comes from the full pass",
|
||||||
|
"p50": 0.05544837899287813,
|
||||||
|
"p95": 0.09468995315801294,
|
||||||
|
"queries": 600
|
||||||
|
},
|
||||||
|
"overall": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.0006830601092896175,
|
||||||
|
"hit_at_10_wilson_lower": 0.00018733970917281023,
|
||||||
|
"mrr": 0.0004098360655737705,
|
||||||
|
"n_answerable": 2928,
|
||||||
|
"n_unanswerable": 2380,
|
||||||
|
"ndcg_at_10": 0.0004736519150391194,
|
||||||
|
"recall_at_1": 0.00034153005464480874,
|
||||||
|
"recall_at_10": 0.0006830601092896175,
|
||||||
|
"recall_at_5": 0.0006830601092896175,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"arm": "no_rerank",
|
||||||
|
"by_modality": {
|
||||||
|
"code": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.5917542441390461,
|
||||||
|
"hit_at_10_wilson_lower": 0.5641209696430517,
|
||||||
|
"mrr": 0.4132014217705406,
|
||||||
|
"n_answerable": 1237,
|
||||||
|
"n_unanswerable": 1237,
|
||||||
|
"ndcg_at_10": 0.4559664301805668,
|
||||||
|
"recall_at_1": 0.3314470493128537,
|
||||||
|
"recall_at_10": 0.5917542441390461,
|
||||||
|
"recall_at_5": 0.52303961196443,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.666127728375101
|
||||||
|
},
|
||||||
|
"conversation": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.54,
|
||||||
|
"hit_at_10_wilson_lower": 0.46023821464666953,
|
||||||
|
"mrr": 0.3229312169312169,
|
||||||
|
"n_answerable": 150,
|
||||||
|
"n_unanswerable": 150,
|
||||||
|
"ndcg_at_10": 0.3349165804130411,
|
||||||
|
"recall_at_1": 0.19,
|
||||||
|
"recall_at_10": 0.44711111111111107,
|
||||||
|
"recall_at_5": 0.38155555555555554,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"image": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.675,
|
||||||
|
"hit_at_10_wilson_lower": 0.6133905629314187,
|
||||||
|
"mrr": 0.5472437169312169,
|
||||||
|
"n_answerable": 240,
|
||||||
|
"n_unanswerable": 240,
|
||||||
|
"ndcg_at_10": 0.5716072495855514,
|
||||||
|
"recall_at_1": 0.4791666666666667,
|
||||||
|
"recall_at_10": 0.66875,
|
||||||
|
"recall_at_5": 0.6,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
},
|
||||||
|
"pdf": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.741514360313316,
|
||||||
|
"hit_at_10_wilson_lower": 0.7093596270841578,
|
||||||
|
"mrr": 0.5952085664553027,
|
||||||
|
"n_answerable": 766,
|
||||||
|
"n_unanswerable": 218,
|
||||||
|
"ndcg_at_10": 0.6306490300771556,
|
||||||
|
"recall_at_1": 0.5195822454308094,
|
||||||
|
"recall_at_10": 0.741514360313316,
|
||||||
|
"recall_at_5": 0.695822454308094,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.6146788990825688
|
||||||
|
},
|
||||||
|
"ui_screenshot": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.296875,
|
||||||
|
"hit_at_10_wilson_lower": 0.24946567415196566,
|
||||||
|
"mrr": 0.1761359126984127,
|
||||||
|
"n_answerable": 320,
|
||||||
|
"n_unanswerable": 320,
|
||||||
|
"ndcg_at_10": 0.20522256852542595,
|
||||||
|
"recall_at_1": 0.11875,
|
||||||
|
"recall_at_10": 0.296875,
|
||||||
|
"recall_at_5": 0.259375,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.009375
|
||||||
|
},
|
||||||
|
"video": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.15348837209302327,
|
||||||
|
"hit_at_10_wilson_lower": 0.11142806397476775,
|
||||||
|
"mrr": 0.0915891472868217,
|
||||||
|
"n_answerable": 215,
|
||||||
|
"n_unanswerable": 215,
|
||||||
|
"ndcg_at_10": 0.10659041518078256,
|
||||||
|
"recall_at_1": 0.06046511627906977,
|
||||||
|
"recall_at_10": 0.15348837209302327,
|
||||||
|
"recall_at_5": 0.13953488372093023,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.004651162790697674
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"latency_seconds": {
|
||||||
|
"max": 10.533955075996346,
|
||||||
|
"mean": 0.9619427272766673,
|
||||||
|
"note": "re-timed on a stride sample with warm models; recall comes from the full pass",
|
||||||
|
"p50": 0.6425201155034301,
|
||||||
|
"p95": 2.786401089788338,
|
||||||
|
"queries": 600
|
||||||
|
},
|
||||||
|
"overall": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.5706967213114754,
|
||||||
|
"hit_at_10_wilson_lower": 0.552686960572952,
|
||||||
|
"mrr": 0.4176547998525457,
|
||||||
|
"n_answerable": 2928,
|
||||||
|
"n_unanswerable": 2380,
|
||||||
|
"ndcg_at_10": 0.4518849109723208,
|
||||||
|
"recall_at_1": 0.34238387978142076,
|
||||||
|
"recall_at_10": 0.5654257741347906,
|
||||||
|
"recall_at_5": 0.510325591985428,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.4042016806722689
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"arm": "fusion_rerank",
|
||||||
|
"by_modality": {
|
||||||
|
"code": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.6289409862570736,
|
||||||
|
"hit_at_10_wilson_lower": 0.6016596375208901,
|
||||||
|
"mrr": 0.4450988695127741,
|
||||||
|
"n_answerable": 1237,
|
||||||
|
"n_unanswerable": 1237,
|
||||||
|
"ndcg_at_10": 0.48964513720914205,
|
||||||
|
"recall_at_1": 0.35084882780921584,
|
||||||
|
"recall_at_10": 0.6289409862570736,
|
||||||
|
"recall_at_5": 0.5675020210185934,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.6313662085691188
|
||||||
|
},
|
||||||
|
"conversation": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.5866666666666667,
|
||||||
|
"hit_at_10_wilson_lower": 0.5066585675662262,
|
||||||
|
"mrr": 0.44420105820105815,
|
||||||
|
"n_answerable": 150,
|
||||||
|
"n_unanswerable": 150,
|
||||||
|
"ndcg_at_10": 0.41993124265105486,
|
||||||
|
"recall_at_1": 0.28,
|
||||||
|
"recall_at_10": 0.4862222222222222,
|
||||||
|
"recall_at_5": 0.42911111111111105,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.006666666666666667
|
||||||
|
},
|
||||||
|
"image": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.7,
|
||||||
|
"hit_at_10_wilson_lower": 0.6392449572501318,
|
||||||
|
"mrr": 0.5878356481481481,
|
||||||
|
"n_answerable": 240,
|
||||||
|
"n_unanswerable": 240,
|
||||||
|
"ndcg_at_10": 0.6098813565843775,
|
||||||
|
"recall_at_1": 0.5208333333333334,
|
||||||
|
"recall_at_10": 0.6916666666666667,
|
||||||
|
"recall_at_5": 0.6520833333333333,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.016666666666666666
|
||||||
|
},
|
||||||
|
"pdf": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.25326370757180156,
|
||||||
|
"hit_at_10_wilson_lower": 0.223750502086736,
|
||||||
|
"mrr": 0.1029870695014298,
|
||||||
|
"n_answerable": 766,
|
||||||
|
"n_unanswerable": 218,
|
||||||
|
"ndcg_at_10": 0.13796388574710278,
|
||||||
|
"recall_at_1": 0.05221932114882506,
|
||||||
|
"recall_at_10": 0.25326370757180156,
|
||||||
|
"recall_at_5": 0.16579634464751958,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0871559633027523
|
||||||
|
},
|
||||||
|
"ui_screenshot": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.3,
|
||||||
|
"hit_at_10_wilson_lower": 0.25240562491350604,
|
||||||
|
"mrr": 0.18429935515873014,
|
||||||
|
"n_answerable": 320,
|
||||||
|
"n_unanswerable": 320,
|
||||||
|
"ndcg_at_10": 0.21181060150559686,
|
||||||
|
"recall_at_1": 0.1375,
|
||||||
|
"recall_at_10": 0.3,
|
||||||
|
"recall_at_5": 0.25625,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.03125
|
||||||
|
},
|
||||||
|
"video": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.08837209302325581,
|
||||||
|
"hit_at_10_wilson_lower": 0.05730442391262212,
|
||||||
|
"mrr": 0.04147840531561462,
|
||||||
|
"n_answerable": 215,
|
||||||
|
"n_unanswerable": 215,
|
||||||
|
"ndcg_at_10": 0.0524917978865275,
|
||||||
|
"recall_at_1": 0.023255813953488372,
|
||||||
|
"recall_at_10": 0.08837209302325581,
|
||||||
|
"recall_at_5": 0.06511627906976744,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"latency_seconds": {
|
||||||
|
"max": 28.03755501300475,
|
||||||
|
"mean": 7.456139394386567,
|
||||||
|
"note": "re-timed on a stride sample with warm models; recall comes from the full pass",
|
||||||
|
"p50": 7.000474203505291,
|
||||||
|
"p95": 13.246865720593268,
|
||||||
|
"queries": 600
|
||||||
|
},
|
||||||
|
"overall": {
|
||||||
|
"abstention_rate_answerable": 0.0,
|
||||||
|
"abstention_rate_unanswerable": 0.0,
|
||||||
|
"hit_at_10": 0.45867486338797814,
|
||||||
|
"hit_at_10_wilson_lower": 0.4406921315699621,
|
||||||
|
"mrr": 0.3091119405412438,
|
||||||
|
"n_answerable": 2928,
|
||||||
|
"n_unanswerable": 2380,
|
||||||
|
"ndcg_at_10": 0.34146096728664366,
|
||||||
|
"recall_at_1": 0.23565573770491804,
|
||||||
|
"recall_at_10": 0.4528460837887068,
|
||||||
|
"recall_at_5": 0.3913479052823315,
|
||||||
|
"top_hit_from_holder_unanswerable": 0.34243697478991597
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"chance": {
|
||||||
|
"corpus_sources": 50278,
|
||||||
|
"recall_at_1": 1.9889414853415013e-05,
|
||||||
|
"recall_at_10": 0.00019889414853415012,
|
||||||
|
"recall_at_5": 9.944707426707506e-05
|
||||||
|
},
|
||||||
|
"live_product": {
|
||||||
|
"distinct_sources_in_top10": {
|
||||||
|
"max": 10,
|
||||||
|
"mean": 7.675,
|
||||||
|
"min": 1
|
||||||
|
},
|
||||||
|
"identical_top10": 0.0,
|
||||||
|
"max": 9.363575160998153,
|
||||||
|
"mean": 1.0766488450000906,
|
||||||
|
"p50": 0.6765933140050038,
|
||||||
|
"queries": 40,
|
||||||
|
"top10_overlap_with_bm25_only": 0.0,
|
||||||
|
"top_k": 10
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"answerable": 2928,
|
||||||
|
"corpus_manifest_sha256": "8fc1ed861c891d778853d41107630869f9eb267a669c5d1038e389de80114347",
|
||||||
|
"corpus_sources": 50278,
|
||||||
|
"generated_at": "2026-10-07T21:24:15+00:00",
|
||||||
|
"git_commit": "dfc6cd4c2889e872d3f7d983b18406e6ff7a622d",
|
||||||
|
"limit_per_modality": null,
|
||||||
|
"machine": {
|
||||||
|
"cpus": 32,
|
||||||
|
"load_average_1_5_15": [
|
||||||
|
1.21,
|
||||||
|
1.55,
|
||||||
|
2.28
|
||||||
|
],
|
||||||
|
"release": "6.18.40.1-microsoft-standard-WSL2",
|
||||||
|
"system": "Linux",
|
||||||
|
"threads_per_query": 4
|
||||||
|
},
|
||||||
|
"models": {
|
||||||
|
"dense": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
||||||
|
"rerank": "BAAI/bge-reranker-base"
|
||||||
|
},
|
||||||
|
"python": "3.14.4",
|
||||||
|
"queries": 5308,
|
||||||
|
"sets": {
|
||||||
|
"code": "ea7e248386350f95132a398f3710f6ce5f09424ca614fd88c618207f667a553e",
|
||||||
|
"conversation": "dc106d90e7749bee8be01b1fab3b645a5459d7527ef40e7fbf983677d5898d11",
|
||||||
|
"image": "1c74488375c7ec9f3182964d51a5e506273bfd5cf3d126725c3cc431a8333128",
|
||||||
|
"pdf": "0cec488e660bf1bfa7ccfc72dbae8b7e3005f0b6a0255028e502296d4ecd4536",
|
||||||
|
"ui_screenshot": "2269dd623e67cd851fb3f6b24a4556a8462a37db98e12ad79577c5830167c7c5",
|
||||||
|
"video": "4973231d9f16d214b14923d79a34b85c3d2b423624dea752b1ee91ba09ff778c"
|
||||||
|
},
|
||||||
|
"store_schema_version_at_start": 5
|
||||||
|
},
|
||||||
|
"scale": [
|
||||||
|
{
|
||||||
|
"atoms": 500,
|
||||||
|
"bm25_only": {
|
||||||
|
"p50": 0.001200532500661211,
|
||||||
|
"p95": 0.0033308441004919584
|
||||||
|
},
|
||||||
|
"dense_only": {
|
||||||
|
"p50": 0.018240716999571305,
|
||||||
|
"p95": 0.03141984814501483
|
||||||
|
},
|
||||||
|
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
|
||||||
|
"queries": 40,
|
||||||
|
"sources": 500
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"atoms": 2000,
|
||||||
|
"bm25_only": {
|
||||||
|
"p50": 0.0047556615027133375,
|
||||||
|
"p95": 0.009782735643602785
|
||||||
|
},
|
||||||
|
"dense_only": {
|
||||||
|
"p50": 0.044618372499826364,
|
||||||
|
"p95": 0.07758056885431869
|
||||||
|
},
|
||||||
|
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
|
||||||
|
"queries": 40,
|
||||||
|
"sources": 2000
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"atoms": 10000,
|
||||||
|
"bm25_only": {
|
||||||
|
"p50": 0.03480406300150207,
|
||||||
|
"p95": 0.11021578034851681
|
||||||
|
},
|
||||||
|
"dense_only": {
|
||||||
|
"p50": 0.07940824350225739,
|
||||||
|
"p95": 0.09990645345169469
|
||||||
|
},
|
||||||
|
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
|
||||||
|
"queries": 40,
|
||||||
|
"sources": 10000
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"atoms": 165695,
|
||||||
|
"bm25_only": {
|
||||||
|
"p50": 0.9441751839985955,
|
||||||
|
"p95": 2.3757928041479315
|
||||||
|
},
|
||||||
|
"dense_only": {
|
||||||
|
"p50": 0.08988298100302927,
|
||||||
|
"p95": 0.10847034014877863
|
||||||
|
},
|
||||||
|
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
|
||||||
|
"queries": 40,
|
||||||
|
"sources": 50000
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
+76
-30
@@ -41,7 +41,7 @@ from bench.systems import (
|
|||||||
ROOT = SETS.parent
|
ROOT = SETS.parent
|
||||||
REPORTS = ROOT / "reports"
|
REPORTS = ROOT / "reports"
|
||||||
DB_HOME = ROOT / "runs" / "baseline"
|
DB_HOME = ROOT / "runs" / "baseline"
|
||||||
LIVE_QUERIES = 40 # the product rebuilds BM25 per query, so a live pass costs minutes per query
|
LIVE_QUERIES = 300 # the acceptance bar names n>=300; the SQL path answers in well under a second
|
||||||
PROGRESS_EVERY = 250 # an arm takes tens of minutes over 5,308 queries; silence looks like a hang
|
PROGRESS_EVERY = 250 # an arm takes tens of minutes over 5,308 queries; silence looks like a hang
|
||||||
SCALE_SIZES = (500, 2_000, 10_000, 50_000)
|
SCALE_SIZES = (500, 2_000, 10_000, 50_000)
|
||||||
SCALE_PROBES = 40
|
SCALE_PROBES = 40
|
||||||
@@ -163,13 +163,19 @@ def live_check(
|
|||||||
top_k: int = 10,
|
top_k: int = 10,
|
||||||
live_queries: int = LIVE_QUERIES,
|
live_queries: int = LIVE_QUERIES,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Compare the unmodified RetrievalService with the fast BM25 path."""
|
"""Score the unmodified RetrievalService on its own, as its holder would ask it.
|
||||||
|
|
||||||
|
The arms measure retrieval configurations; the acceptance bar is about the product. A query is
|
||||||
|
asked as the holder the frozen set assigns it, because a transactive memory is used that way.
|
||||||
|
"""
|
||||||
probe = [query for query in queries if query.answerable]
|
probe = [query for query in queries if query.answerable]
|
||||||
stride = max(1, len(probe) // live_queries)
|
stride = max(1, len(probe) // live_queries)
|
||||||
chosen = probe[::stride][:live_queries]
|
chosen = probe[::stride][:live_queries]
|
||||||
if not chosen:
|
if not chosen:
|
||||||
raise SystemExit("live check needs answerable queries")
|
raise SystemExit("live check needs answerable queries")
|
||||||
rankings, distinct, timings = live_product_rankings(DB_HOME, [query.text for query in chosen], top_k)
|
rankings, distinct, timings = live_product_rankings(
|
||||||
|
DB_HOME, [query.text for query in chosen], top_k, as_holders=[query.holder for query in chosen]
|
||||||
|
)
|
||||||
overlaps = [
|
overlaps = [
|
||||||
len(set(fast_rankings[query.query_id]) & set(ranking)) / top_k
|
len(set(fast_rankings[query.query_id]) & set(ranking)) / top_k
|
||||||
for query, ranking in zip(chosen, rankings, strict=True)
|
for query, ranking in zip(chosen, rankings, strict=True)
|
||||||
@@ -177,20 +183,30 @@ def live_check(
|
|||||||
]
|
]
|
||||||
if not overlaps:
|
if not overlaps:
|
||||||
raise SystemExit("live check found no fast rankings to compare against")
|
raise SystemExit("live check found no fast rankings to compare against")
|
||||||
return {
|
scores = [
|
||||||
"queries": len(chosen),
|
score_query(query.query_id, ranking, set(query.gold), query.holder)
|
||||||
"top_k": top_k,
|
for query, ranking in zip(chosen, rankings, strict=True)
|
||||||
"p50": sorted(timings)[len(timings) // 2],
|
]
|
||||||
"mean": sum(timings) / len(timings),
|
metrics: dict[str, Any] = dict(aggregate(scores))
|
||||||
"max": max(timings),
|
ordered = sorted(timings)
|
||||||
"top10_overlap_with_bm25_only": sum(overlaps) / len(overlaps),
|
metrics.update(
|
||||||
"identical_top10": sum(1 for overlap in overlaps if overlap == 1.0) / len(overlaps),
|
{
|
||||||
"distinct_sources_in_top10": {
|
"queries": len(chosen),
|
||||||
"mean": sum(distinct) / len(distinct),
|
"top_k": top_k,
|
||||||
"min": min(distinct),
|
"p50": ordered[len(ordered) // 2],
|
||||||
"max": max(distinct),
|
"p95": ordered[min(len(ordered) - 1, int(len(ordered) * 0.95))],
|
||||||
},
|
"mean": sum(ordered) / len(ordered),
|
||||||
}
|
"max": max(ordered),
|
||||||
|
"top10_overlap_with_bm25_only": sum(overlaps) / len(overlaps),
|
||||||
|
"identical_top10": sum(1 for overlap in overlaps if overlap == 1.0) / len(overlaps),
|
||||||
|
"distinct_sources_in_top10": {
|
||||||
|
"mean": sum(distinct) / len(distinct),
|
||||||
|
"min": min(distinct),
|
||||||
|
"max": max(distinct),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return metrics
|
||||||
|
|
||||||
|
|
||||||
def git_commit() -> str:
|
def git_commit() -> str:
|
||||||
@@ -203,7 +219,23 @@ def git_commit() -> str:
|
|||||||
return commit
|
return commit
|
||||||
|
|
||||||
|
|
||||||
def meta(corpus_sources: int, queries: Sequence[Query], limit: int | None = None) -> dict[str, Any]:
|
def store_schema_version(store: Path) -> int:
|
||||||
|
"""The schema the measured store actually carries, read from the file."""
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
if not store.exists():
|
||||||
|
raise SystemExit(f"{store}: the measured store is missing")
|
||||||
|
connection = sqlite3.connect(f"file:{store}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
row = connection.execute("PRAGMA user_version").fetchone()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
return int(row[0])
|
||||||
|
|
||||||
|
|
||||||
|
def meta(
|
||||||
|
corpus_sources: int, queries: Sequence[Query], limit: int | None = None, store: Path | None = None
|
||||||
|
) -> dict[str, Any]:
|
||||||
manifest_path = SETS / "manifest.json"
|
manifest_path = SETS / "manifest.json"
|
||||||
try:
|
try:
|
||||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
@@ -220,6 +252,7 @@ def meta(corpus_sources: int, queries: Sequence[Query], limit: int | None = None
|
|||||||
"answerable": sum(1 for query in queries if query.answerable),
|
"answerable": sum(1 for query in queries if query.answerable),
|
||||||
"limit_per_modality": limit,
|
"limit_per_modality": limit,
|
||||||
"corpus_sources": corpus_sources,
|
"corpus_sources": corpus_sources,
|
||||||
|
"store_schema_version_at_start": None if store is None else store_schema_version(store),
|
||||||
"models": {"dense": DENSE_MODEL, "rerank": RERANK_MODEL},
|
"models": {"dense": DENSE_MODEL, "rerank": RERANK_MODEL},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -240,12 +273,14 @@ def scale_arm_names(rows: Sequence[dict[str, Any]]) -> list[str]:
|
|||||||
return names
|
return names
|
||||||
|
|
||||||
|
|
||||||
def write_markdown(report: dict[str, Any], path: Path) -> None:
|
def write_markdown(report: dict[str, Any], path: Path, name: str = "BASELINE") -> None:
|
||||||
"""Human-readable twin of the JSON report."""
|
"""Human-readable twin of the JSON report."""
|
||||||
|
schema = report["meta"].get("store_schema_version_at_start")
|
||||||
lines = [
|
lines = [
|
||||||
"# BASELINE: what the product retrieves today",
|
f"# {name}: what the product retrieves at this build",
|
||||||
"",
|
"",
|
||||||
f"- generated {report['meta']['generated_at']} at commit {report['meta']['git_commit'][:12]}",
|
f"- generated {report['meta']['generated_at']} at commit {report['meta']['git_commit'][:12]}",
|
||||||
|
("" if schema is None else f"- measured store schema: v{schema}"),
|
||||||
(
|
(
|
||||||
f"- corpus {report['meta']['corpus_sources']} sources, {report['meta']['queries']} queries "
|
f"- corpus {report['meta']['corpus_sources']} sources, {report['meta']['queries']} queries "
|
||||||
f"({report['meta']['answerable']} answerable)"
|
f"({report['meta']['answerable']} answerable)"
|
||||||
@@ -302,21 +337,26 @@ def write_markdown(report: dict[str, Any], path: Path) -> None:
|
|||||||
live = report["live_product"]
|
live = report["live_product"]
|
||||||
lines += [
|
lines += [
|
||||||
"",
|
"",
|
||||||
"## the unmodified product path",
|
"## the product path, asked as its holder",
|
||||||
"",
|
"",
|
||||||
(
|
(
|
||||||
f"- RetrievalService.search: p50 {live['p50']:.2f}s, mean {live['mean']:.2f}s, max {live['max']:.2f}s "
|
f"- RetrievalService.search as the query's own holder: recall@10 {live['recall_at_10']:.3f}, "
|
||||||
f"over {live['queries']} queries"
|
f"MRR {live['mrr']:.3f}, hit@10 {live['hit_at_10']:.3f} (95% lower {live['hit_at_10_wilson_lower']:.3f}) "
|
||||||
|
f"over n={live['queries']} answerable queries"
|
||||||
|
),
|
||||||
|
(
|
||||||
|
f"- latency p50 {live['p50']:.3f}s, p95 {live['p95']:.3f}s, mean {live['mean']:.3f}s, "
|
||||||
|
f"max {live['max']:.3f}s"
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
f"- top-10 overlap with the fast BM25 path: {live['top10_overlap_with_bm25_only']:.1%}, "
|
f"- top-10 overlap with the fast BM25 path: {live['top10_overlap_with_bm25_only']:.1%}, "
|
||||||
f"identical order {live['identical_top10']:.1%}"
|
f"identical order {live['identical_top10']:.1%}"
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
f"- its top-{live['top_k']} atoms cover only "
|
f"- its top-{live['top_k']} atoms cover "
|
||||||
f"{live['distinct_sources_in_top10']['mean']:.1f} distinct sources "
|
f"{live['distinct_sources_in_top10']['mean']:.1f} distinct sources on average "
|
||||||
f"(min {live['distinct_sources_in_top10']['min']}): the product ranks atoms and never diversifies, "
|
f"(min {live['distinct_sources_in_top10']['min']}): the product ranks atoms and never diversifies, "
|
||||||
"so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero"
|
"so ten slots can fill with fragments of one document"
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
if report.get("scale"):
|
if report.get("scale"):
|
||||||
@@ -463,6 +503,11 @@ def main() -> int:
|
|||||||
help="re-time this many sampled queries with warm models instead of trusting cached per-query seconds",
|
help="re-time this many sampled queries with warm models instead of trusting cached per-query seconds",
|
||||||
)
|
)
|
||||||
parser.add_argument("--out", type=Path, default=REPORTS)
|
parser.add_argument("--out", type=Path, default=REPORTS)
|
||||||
|
parser.add_argument(
|
||||||
|
"--name",
|
||||||
|
default="BASELINE",
|
||||||
|
help="report name: writes NAME.md and name.json (BASELINE stays the frozen pre-P0 snapshot)",
|
||||||
|
)
|
||||||
parser.add_argument("--config", type=Path, help="JSON defaults for these flags (bench/configs/ablation.json)")
|
parser.add_argument("--config", type=Path, help="JSON defaults for these flags (bench/configs/ablation.json)")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
if args.config:
|
if args.config:
|
||||||
@@ -485,7 +530,7 @@ def main() -> int:
|
|||||||
print(f"{len(queries)} queries, {len(atoms)} atoms, {corpus_sources} sources", flush=True)
|
print(f"{len(queries)} queries, {len(atoms)} atoms, {corpus_sources} sources", flush=True)
|
||||||
|
|
||||||
report: dict[str, Any] = {
|
report: dict[str, Any] = {
|
||||||
"meta": meta(corpus_sources, queries, args.limit),
|
"meta": meta(corpus_sources, queries, args.limit, store=DB_HOME / "data" / "ely_eye.sqlite"),
|
||||||
"chance": chance_row(queries, corpus_sources),
|
"chance": chance_row(queries, corpus_sources),
|
||||||
}
|
}
|
||||||
# One pass over the queries: each signal is computed once and every arm is derived
|
# One pass over the queries: each signal is computed once and every arm is derived
|
||||||
@@ -553,9 +598,10 @@ def main() -> int:
|
|||||||
report["scale"] = scale_curve(queries, atoms, [arm for arm in arms_requested if arm in ("bm25_only", "dense_only")])
|
report["scale"] = scale_curve(queries, atoms, [arm for arm in arms_requested if arm in ("bm25_only", "dense_only")])
|
||||||
|
|
||||||
args.out.mkdir(parents=True, exist_ok=True)
|
args.out.mkdir(parents=True, exist_ok=True)
|
||||||
(args.out / "baseline.json").write_text(json.dumps(report, indent=1, sort_keys=True), encoding="utf-8")
|
stem = args.name.lower()
|
||||||
write_markdown(report, args.out / "BASELINE.md")
|
(args.out / f"{stem}.json").write_text(json.dumps(report, indent=1, sort_keys=True), encoding="utf-8")
|
||||||
print(f"== wrote {args.out / 'baseline.json'} and {args.out / 'BASELINE.md'}", flush=True)
|
write_markdown(report, args.out / f"{args.name}.md", name=args.name)
|
||||||
|
print(f"== wrote {args.out / (stem + '.json')} and {args.out / (args.name + '.md')}", flush=True)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,78 @@
|
|||||||
|
"""Give every ingested atom the holder, scenario and event time the corpus index already states.
|
||||||
|
|
||||||
|
The pre-P0 store was built before the memory model had those columns, so it holds 165,973 memories
|
||||||
|
nobody owns. The frozen corpus index names the owner, the setting and the moment for each source —
|
||||||
|
the same facts the counterpart and the gold scopes are written from — so the store can be brought
|
||||||
|
in line without re-running ingestion.
|
||||||
|
|
||||||
|
Run: python -m bench.stamp_holders [--store bench/runs/baseline/data/ely_eye.sqlite]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from bench.sets_common import MODALITIES, SETS, read_jsonl, record_path
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parent.parent
|
||||||
|
DEFAULT_STORE = REPO / "bench" / "runs" / "baseline" / "data" / "ely_eye.sqlite"
|
||||||
|
|
||||||
|
|
||||||
|
def corpus_facts() -> dict[str, tuple[str, str, str]]:
|
||||||
|
"""Absolute path -> (holder, scenario, observed_at), as the frozen corpus index states them."""
|
||||||
|
facts: dict[str, tuple[str, str, str]] = {}
|
||||||
|
for modality in MODALITIES:
|
||||||
|
for record in read_jsonl(SETS / "corpus" / f"{modality}.jsonl"):
|
||||||
|
for key in ("holder", "scenario", "observed_at"):
|
||||||
|
if not record.get(key):
|
||||||
|
raise SystemExit(f"{modality}: corpus record {record['source_id']} has no {key}")
|
||||||
|
facts[str(record_path(record).resolve())] = (
|
||||||
|
str(record["holder"]),
|
||||||
|
str(record["scenario"]),
|
||||||
|
str(record["observed_at"]),
|
||||||
|
)
|
||||||
|
if not facts:
|
||||||
|
raise SystemExit("no corpus index found, run python -m bench.build_sets first")
|
||||||
|
return facts
|
||||||
|
|
||||||
|
|
||||||
|
def stamp(store: Path) -> dict[str, int]:
|
||||||
|
"""Write the corpus facts onto every atom; the store's own source ids stay untouched."""
|
||||||
|
if not store.is_file():
|
||||||
|
raise SystemExit(f"{store}: no baseline database, run python -m bench.ingest_baseline first")
|
||||||
|
facts = corpus_facts()
|
||||||
|
with sqlite3.connect(store) as connection:
|
||||||
|
rows = connection.execute("SELECT source_id, path FROM sources").fetchall()
|
||||||
|
if not rows:
|
||||||
|
raise SystemExit(f"{store}: the sources table is empty")
|
||||||
|
unknown = sorted(str(path) for _, path in rows if str(path) not in facts)
|
||||||
|
if unknown:
|
||||||
|
raise SystemExit(f"{store}: {len(unknown)} source paths are not corpus files, first {unknown[:3]}")
|
||||||
|
stamped_atoms = 0
|
||||||
|
for source_id, path in rows:
|
||||||
|
holder, scenario, observed_at = facts[str(path)]
|
||||||
|
cursor = connection.execute(
|
||||||
|
"UPDATE atoms SET holder_id = ?, scenario = ?, event_time = ?, event_time_precision = 'instant' "
|
||||||
|
"WHERE source_id = ?",
|
||||||
|
(holder, scenario, observed_at, str(source_id)),
|
||||||
|
)
|
||||||
|
stamped_atoms += int(cursor.rowcount)
|
||||||
|
left = int(connection.execute("SELECT count(*) FROM atoms WHERE holder_id = ''").fetchone()[0])
|
||||||
|
return {"sources": len(rows), "atoms": stamped_atoms, "atoms_without_holder": left}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--store", type=Path, default=DEFAULT_STORE)
|
||||||
|
args = parser.parse_args()
|
||||||
|
counts = stamp(args.store)
|
||||||
|
print(
|
||||||
|
f"stamped {counts['atoms']} atoms across {counts['sources']} sources; "
|
||||||
|
f"{counts['atoms_without_holder']} atoms still have no holder"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+46
-10
@@ -18,11 +18,14 @@ import time
|
|||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from bench.ingest_baseline import source_map
|
from bench.ingest_baseline import source_map
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from ely_eye.db import Database
|
||||||
from bench.sets_common import ROOT
|
from bench.sets_common import ROOT
|
||||||
|
|
||||||
RUNS = ROOT / "runs" / "baseline"
|
RUNS = ROOT / "runs" / "baseline"
|
||||||
@@ -423,34 +426,67 @@ def _best_atom_text(atoms: Sequence[Atom], source_ids: Sequence[str]) -> list[tu
|
|||||||
return [(source_id, longest.get(source_id, "")) for source_id in source_ids]
|
return [(source_id, longest.get(source_id, "")) for source_id in source_ids]
|
||||||
|
|
||||||
|
|
||||||
|
def _corpus_ids(database: Database) -> dict[str, str]:
|
||||||
|
"""Join the store's content-hash source ids to the ids the frozen sets call a source by.
|
||||||
|
|
||||||
|
Without this join the product's hits and the gold answers share no name, and a correct answer
|
||||||
|
scores zero — which is what an earlier report recorded as "0% overlap with the fast path".
|
||||||
|
"""
|
||||||
|
mapping = source_map()
|
||||||
|
ids: dict[str, str] = {}
|
||||||
|
unknown: list[str] = []
|
||||||
|
for source in database.list_sources():
|
||||||
|
corpus_id = mapping.get(str(Path(source.path).resolve()))
|
||||||
|
if corpus_id is None:
|
||||||
|
unknown.append(source.source_id)
|
||||||
|
else:
|
||||||
|
ids[source.source_id] = corpus_id
|
||||||
|
if unknown:
|
||||||
|
raise SystemExit(f"{database.path}: {len(unknown)} store sources are not corpus files, first {unknown[:3]}")
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
def live_product_rankings(
|
def live_product_rankings(
|
||||||
db_home: Path, queries: Sequence[str], top_k: int = 10
|
db_home: Path,
|
||||||
|
queries: Sequence[str],
|
||||||
|
top_k: int = 10,
|
||||||
|
as_holders: Sequence[str | None] | None = None,
|
||||||
) -> tuple[list[list[str]], list[int], list[float]]:
|
) -> tuple[list[list[str]], list[int], list[float]]:
|
||||||
"""Rankings from the unmodified RetrievalService, with per-query wall time.
|
"""Rankings from the unmodified RetrievalService, with per-query wall time.
|
||||||
|
|
||||||
The second list counts how many distinct sources the product's top-k atoms came
|
Rankings carry the benchmark's source ids, so they are comparable with the arms and with gold.
|
||||||
from: the product ranks atoms and never diversifies, so ten slots can be filled by
|
`as_holders` asks each query as its own holder, which is how a transactive memory is used: the
|
||||||
fragments of one document, and that is invisible once the hits are collapsed.
|
asker does not know where the answer is, only who might.
|
||||||
|
|
||||||
|
The second list counts how many distinct sources the product's top-k atoms came from: the
|
||||||
|
product ranks atoms and never diversifies, so ten slots can be filled by fragments of one
|
||||||
|
document, and that is invisible once the hits are collapsed.
|
||||||
"""
|
"""
|
||||||
from ely_eye.config import Settings
|
from ely_eye.config import Settings
|
||||||
from ely_eye.db import Database
|
from ely_eye.db import Database
|
||||||
from ely_eye.retrieval import RetrievalService
|
from ely_eye.retrieval import RetrievalService
|
||||||
|
from ely_eye.schemas import SearchFilters
|
||||||
|
|
||||||
|
if as_holders is not None and len(as_holders) != len(queries):
|
||||||
|
raise SystemExit(f"{len(as_holders)} holders for {len(queries)} queries")
|
||||||
settings = Settings(home=db_home)
|
settings = Settings(home=db_home)
|
||||||
service = RetrievalService(settings=settings, database=Database(settings))
|
database = Database(settings)
|
||||||
|
service = RetrievalService(settings=settings, database=database)
|
||||||
|
corpus_ids = _corpus_ids(database)
|
||||||
rankings: list[list[str]] = []
|
rankings: list[list[str]] = []
|
||||||
distinct: list[int] = []
|
distinct: list[int] = []
|
||||||
timings: list[float] = []
|
timings: list[float] = []
|
||||||
for query in queries:
|
for index, query in enumerate(queries):
|
||||||
|
holder = as_holders[index] if as_holders is not None else None
|
||||||
started = time.perf_counter()
|
started = time.perf_counter()
|
||||||
hits = service.search(query, top_k=top_k).hits
|
hits = service.search(query, top_k=top_k, filters=SearchFilters(as_holder=holder)).hits
|
||||||
timings.append(time.perf_counter() - started)
|
timings.append(time.perf_counter() - started)
|
||||||
# The product ranks atoms; collapse to sources by first (best) hit so the arms stay comparable.
|
# The product ranks atoms; collapse to sources by first (best) hit so the arms stay comparable.
|
||||||
sources: list[str] = []
|
sources: list[str] = []
|
||||||
for hit in hits:
|
for hit in hits:
|
||||||
source_id = hit.atom.source_id
|
source_id = corpus_ids[hit.atom.source_id]
|
||||||
if source_id not in sources:
|
if source_id not in sources:
|
||||||
sources.append(source_id)
|
sources.append(source_id)
|
||||||
distinct.append(len({hit.atom.source_id for hit in hits}))
|
distinct.append(len({corpus_ids[hit.atom.source_id] for hit in hits}))
|
||||||
rankings.append(sources[:top_k])
|
rankings.append(sources[:top_k])
|
||||||
return rankings, distinct, timings
|
return rankings, distinct, timings
|
||||||
@@ -181,7 +181,12 @@ def _report() -> dict:
|
|||||||
"live_product": {
|
"live_product": {
|
||||||
"queries": 2,
|
"queries": 2,
|
||||||
"top_k": 10,
|
"top_k": 10,
|
||||||
|
"recall_at_10": 0.87,
|
||||||
|
"mrr": 0.74,
|
||||||
|
"hit_at_10": 0.87,
|
||||||
|
"hit_at_10_wilson_lower": 0.82,
|
||||||
"p50": 1.0,
|
"p50": 1.0,
|
||||||
|
"p95": 1.0,
|
||||||
"mean": 1.0,
|
"mean": 1.0,
|
||||||
"max": 1.0,
|
"max": 1.0,
|
||||||
"top10_overlap_with_bm25_only": 0.0,
|
"top10_overlap_with_bm25_only": 0.0,
|
||||||
@@ -202,6 +207,7 @@ def test_the_report_writer_survives_a_scale_note(tmp_path: Path) -> None:
|
|||||||
assert "_latency only_" in text
|
assert "_latency only_" in text
|
||||||
assert "load average 1.0 on 32 cores" in text
|
assert "load average 1.0 on 32 cores" in text
|
||||||
assert "## unanswerable queries (50.0% of the suite)" in text
|
assert "## unanswerable queries (50.0% of the suite)" in text
|
||||||
|
assert "recall@10 0.870" in text
|
||||||
|
|
||||||
|
|
||||||
def test_scale_arm_names_reads_only_latency_dicts() -> None:
|
def test_scale_arm_names_reads_only_latency_dicts() -> None:
|
||||||
|
|||||||
Reference in new issue
Block a user