ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give every memory a holder, an origin, a time and a scenario

Browse Source 
An EvidenceAtom could say what was remembered and nothing else: not whose memory it
is, not where it came from, not when it was true, not in what situation it may be
used. The baseline measured the cost of that: the same ranker scores recall@10 0.412
over the whole corpus and 0.866 inside one holder's memory, so the missing holder
column is worth more than any ranking change in the ablation.

- schemas.py: MemoryOrigin, TimePrecision, Visibility; EvidenceAtom gains holder_id,
  origin, scenario, event_time / event_time_end / event_time_precision /
  event_time_source, valid_from / valid_to, supersedes, visibility, retention_until.
  `time` is renamed `ingested_at` — ingestion time gets one name, not two.
  A validator refuses valid_to before valid_from, event_time_end before event_time,
  and an atom that supersedes itself.
- migrations.py: schema v2 with PRAGMA user_version. A first-version database is
  upgraded in place (ALTER + RENAME COLUMN), rows intact, and a database newer than
  this build refuses to open instead of being silently downgraded.
- v_atoms_current hides memories that are closed, lapsed or not yet begun; unknown
  bounds read as visible. Time travel is db.atoms_as_of(moment, holder) because a
  SQLite view cannot take a parameter.
- evaluation_schemas.py: the proof and benchmark report shapes move out of schemas.py
  to keep it under the file-size limit; importers follow.
- backend/tests/test_memory_schema.py: 8 tests, including the v1 -> v2 upgrade and the
  validity timeline.

Verification of record: pytest backend/tests bench/tests -q -> 70 passed; ruff check on
every touched file -> all checks passed; mypy (fresh cache) on db.py, migrations.py,
schemas.py, evaluation_schemas.py and backend/tests -> no issues. evals.py carries 40
pre-existing mypy errors at HEAD and 40 after this change (measured on both copies);
its pyright findings are the same debt, and p0-6 rewrites that file. `peft` and
`pyarrow.parquet` "could not be resolved" is the documented pyright artifact: pi-lens
runs it with 0 config files, and both import at runtime.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 14:30:22 -04:00
1 parent 00dc9f8eaf
commit 181f6bf57e
12 files changed
+1100 -319
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bench/reports/BASELINE_EXTERNAL.md
+64
View File
@@ -0,0 +1,64 @@
# BASELINE_EXTERNAL: memory counterparts on the same corpus
- generated 2026-10-07T17:26:07+00:00 at commit c4e2982506fa
- 5308 queries (2928 answerable) against 50789 sources
- machine: 32 cores, load average 1.5 when the report was written
- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have
| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | abstain (answerable) | abstain (unanswerable) | p50 s | p95 s |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mem0_flat [holder scope] | 403859 | 0 | 0.866 | 0.867 | 0.855 | 0.629 | 0.000 | 0.000 | 0.084 | 0.133 |
| mem0_flat [whole corpus] | 403859 | 0 | 0.412 | 0.416 | 0.398 | 0.283 | 0.000 | 0.000 | 0.090 | 0.174 |
| zep_temporal [holder scope] | 403859 | 27072 | 0.866 | 0.867 | 0.854 | 0.627 | 0.000 | 0.058 | 0.088 | 0.111 |
| zep_temporal [whole corpus] | 403859 | 27072 | 0.429 | 0.433 | 0.415 | 0.297 | 0.000 | 0.000 | 0.086 | 0.107 |
n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}
`[holder scope]` answers only from the asking holder's memory; `[whole corpus]` searches every
holder's memory, which is what the product's arms are shown. The gap between the two rows is the
value of the holder boundary, not of a better ranker.
## mem0_flat [holder scope] by modality
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
| --- | --- | --- | --- | --- | --- |
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
| conversation | 150 | 0.993 | 1.000 | 0.975 | 0.927 |
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
## mem0_flat [whole corpus] by modality
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
| --- | --- | --- | --- | --- | --- |
| code | 1237 | 0.349 | 0.349 | 0.323 | 0.216 |
| conversation | 150 | 0.500 | 0.573 | 0.493 | 0.373 |
| image | 240 | 0.525 | 0.533 | 0.470 | 0.362 |
| pdf | 766 | 0.683 | 0.683 | 0.649 | 0.515 |
| ui_screenshot | 320 | 0.075 | 0.075 | 0.051 | 0.033 |
| video | 215 | 0.116 | 0.116 | 0.080 | 0.069 |
## zep_temporal [holder scope] by modality
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
| --- | --- | --- | --- | --- | --- |
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
| conversation | 150 | 0.978 | 0.987 | 0.953 | 0.905 |
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
## zep_temporal [whole corpus] by modality
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
| --- | --- | --- | --- | --- | --- |
| code | 1237 | 0.350 | 0.350 | 0.324 | 0.216 |
| conversation | 150 | 0.591 | 0.667 | 0.588 | 0.429 |
| image | 240 | 0.562 | 0.571 | 0.508 | 0.426 |
| pdf | 766 | 0.685 | 0.685 | 0.652 | 0.519 |
| ui_screenshot | 320 | 0.119 | 0.119 | 0.088 | 0.057 |
| video | 215 | 0.163 | 0.163 | 0.119 | 0.087 |
bench/reports/baseline_external.json
+252 -28
View File
@@ -2,15 +2,15 @@
"meta": {
"answerable": 2928,
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
"generated_at": "2026-10-07T16:54:23+00:00",
"generated_at": "2026-10-07T17:26:07+00:00",
"git_commit": "c4e2982506faaf350a4b06738923899fc44bdedd",
"limit_per_modality": null,
"machine": {
"cpus": 32,
"load_average_1_5_15": [
6.23,
4.72,
4.44
1.47,
2.81,
10.33
],
"release": "6.18.40.1-microsoft-standard-WSL2",
"system": "Linux"
@@ -42,7 +42,7 @@
"recall_at_1": 0.549717057396928,
"recall_at_10": 0.952303961196443,
"recall_at_5": 0.8884397736459175,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"conversation": {
"abstention_rate_answerable": 0.0,
@@ -56,7 +56,7 @@
"recall_at_1": 0.657888888888889,
"recall_at_10": 0.9927777777777778,
"recall_at_5": 0.9814444444444445,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"image": {
"abstention_rate_answerable": 0.0,
@@ -70,7 +70,7 @@
"recall_at_1": 0.42291666666666666,
"recall_at_10": 0.6458333333333334,
"recall_at_5": 0.6,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"pdf": {
"abstention_rate_answerable": 0.0,
@@ -84,7 +84,7 @@
"recall_at_1": 0.587467362924282,
"recall_at_10": 0.9164490861618799,
"recall_at_5": 0.825065274151436,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"ui_screenshot": {
"abstention_rate_answerable": 0.0,
@@ -98,7 +98,7 @@
"recall_at_1": 0.21875,
"recall_at_10": 0.621875,
"recall_at_5": 0.465625,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"video": {
"abstention_rate_answerable": 0.0,
@@ -112,16 +112,16 @@
"recall_at_1": 0.2837209302325581,
"recall_at_10": 0.7162790697674418,
"recall_at_5": 0.5255813953488372,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
}
},
"closed_items": 0,
"items": 403859,
"latency_seconds": {
"max": 15.670968882004672,
"mean": 0.09444417075096513,
"p50": 0.08692938649983262,
"p95": 0.10732038790265505,
"max": 7.547149860001809,
"mean": 0.09187759584445908,
"p50": 0.08432659949903609,
"p95": 0.13331470334669573,
"queries": 5308
},
"overall": {
@@ -136,9 +136,121 @@
"recall_at_1": 0.49903802367941713,
"recall_at_10": 0.8664332877959927,
"recall_at_5": 0.7801286429872496,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"store": "mem0_flat"
"store": "mem0_flat [holder scope]"
},
{
"by_modality": {
"code": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.349232012934519,
"hit_at_10_wilson_lower": 0.3231693767503265,
"mrr": 0.21579377911229164,
"n_answerable": 1237,
"n_unanswerable": 1237,
"ndcg_at_10": 0.24771932445387607,
"recall_at_1": 0.15683104284559418,
"recall_at_10": 0.349232012934519,
"recall_at_5": 0.2991107518189167,
"top_hit_from_holder_unanswerable": 0.0
},
"conversation": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.5733333333333334,
"hit_at_10_wilson_lower": 0.4933252687249978,
"mrr": 0.37327513227513226,
"n_answerable": 150,
"n_unanswerable": 150,
"ndcg_at_10": 0.38064109675098423,
"recall_at_1": 0.2188888888888889,
"recall_at_10": 0.5004444444444445,
"recall_at_5": 0.4146666666666667,
"top_hit_from_holder_unanswerable": 0.0
},
"image": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.5333333333333333,
"hit_at_10_wilson_lower": 0.4701883065323697,
"mrr": 0.36166170634920636,
"n_answerable": 240,
"n_unanswerable": 240,
"ndcg_at_10": 0.3979762690192957,
"recall_at_1": 0.26458333333333334,
"recall_at_10": 0.525,
"recall_at_5": 0.4708333333333333,
"top_hit_from_holder_unanswerable": 0.0
},
"pdf": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.6827676240208878,
"hit_at_10_wilson_lower": 0.6489674562308053,
"mrr": 0.5150985328857391,
"n_answerable": 766,
"n_unanswerable": 218,
"ndcg_at_10": 0.555124937448977,
"recall_at_1": 0.43994778067885115,
"recall_at_10": 0.6827676240208878,
"recall_at_5": 0.6109660574412533,
"top_hit_from_holder_unanswerable": 0.0
},
"ui_screenshot": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.075,
"hit_at_10_wilson_lower": 0.05091491480619657,
"mrr": 0.032880704365079366,
"n_answerable": 320,
"n_unanswerable": 320,
"ndcg_at_10": 0.04277852579814079,
"recall_at_1": 0.01875,
"recall_at_10": 0.075,
"recall_at_5": 0.059375,
"top_hit_from_holder_unanswerable": 0.0
},
"video": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.11627906976744186,
"hit_at_10_wilson_lower": 0.08001303523895324,
"mrr": 0.06852713178294573,
"n_answerable": 215,
"n_unanswerable": 215,
"ndcg_at_10": 0.08038135054972745,
"recall_at_1": 0.04186046511627907,
"recall_at_10": 0.11627906976744186,
"recall_at_5": 0.11627906976744186,
"top_hit_from_holder_unanswerable": 0.0
}
},
"closed_items": 0,
"items": 403859,
"latency_seconds": {
"max": 2.318251260003308,
"mean": 0.10140667538014644,
"p50": 0.09026374550012406,
"p95": 0.17443375029943126,
"queries": 5308
},
"overall": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.41598360655737704,
"hit_at_10_wilson_lower": 0.39825196843019733,
"mrr": 0.2833154436638043,
"n_answerable": 2928,
"n_unanswerable": 2380,
"ndcg_at_10": 0.31258063324325003,
"recall_at_1": 0.2193761384335155,
"recall_at_10": 0.41156648451730415,
"recall_at_5": 0.3610655737704918,
"top_hit_from_holder_unanswerable": 0.0
},
"store": "mem0_flat [whole corpus]"
},
{
"by_modality": {
@@ -154,7 +266,7 @@
"recall_at_1": 0.549717057396928,
"recall_at_10": 0.952303961196443,
"recall_at_5": 0.8884397736459175,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"conversation": {
"abstention_rate_answerable": 0.0,
@@ -168,7 +280,7 @@
"recall_at_1": 0.637888888888889,
"recall_at_10": 0.9783333333333334,
"recall_at_5": 0.9458888888888889,
"wrong_holder_confidence_unanswerable": 0.8866666666666667
"top_hit_from_holder_unanswerable": 0.8866666666666667
},
"image": {
"abstention_rate_answerable": 0.0,
@@ -182,7 +294,7 @@
"recall_at_1": 0.42291666666666666,
"recall_at_10": 0.6458333333333334,
"recall_at_5": 0.6,
"wrong_holder_confidence_unanswerable": 0.5
"top_hit_from_holder_unanswerable": 0.5
},
"pdf": {
"abstention_rate_answerable": 0.0,
@@ -196,7 +308,7 @@
"recall_at_1": 0.587467362924282,
"recall_at_10": 0.9164490861618799,
"recall_at_5": 0.825065274151436,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"ui_screenshot": {
"abstention_rate_answerable": 0.0,
@@ -210,7 +322,7 @@
"recall_at_1": 0.21875,
"recall_at_10": 0.621875,
"recall_at_5": 0.465625,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
},
"video": {
"abstention_rate_answerable": 0.0,
@@ -224,16 +336,16 @@
"recall_at_1": 0.2837209302325581,
"recall_at_10": 0.7162790697674418,
"recall_at_5": 0.5255813953488372,
"wrong_holder_confidence_unanswerable": 1.0
"top_hit_from_holder_unanswerable": 1.0
}
},
"closed_items": 27072,
"items": 403859,
"latency_seconds": {
"max": 10.518811015004758,
"mean": 0.09140969146306362,
"p50": 0.0847548475030635,
"p95": 0.10844126994816179,
"max": 18.72793256899604,
"mean": 0.09218696485981677,
"p50": 0.08754261549984221,
"p95": 0.11125843500121846,
"queries": 5308
},
"overall": {
@@ -248,9 +360,121 @@
"recall_at_1": 0.4980134335154827,
"recall_at_10": 0.865693306010929,
"recall_at_5": 0.7783071493624771,
"wrong_holder_confidence_unanswerable": 0.942436974789916
"top_hit_from_holder_unanswerable": 0.942436974789916
},
"store": "zep_temporal"
"store": "zep_temporal [holder scope]"
},
{
"by_modality": {
"code": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.35004042037186744,
"hit_at_10_wilson_lower": 0.32396114426502914,
"mrr": 0.21595449820995496,
"n_answerable": 1237,
"n_unanswerable": 1237,
"ndcg_at_10": 0.24802826744611478,
"recall_at_1": 0.15683104284559418,
"recall_at_10": 0.35004042037186744,
"recall_at_5": 0.2991107518189167,
"top_hit_from_holder_unanswerable": 0.0
},
"conversation": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.6666666666666666,
"hit_at_10_wilson_lower": 0.5878975668678504,
"mrr": 0.42928042328042326,
"n_answerable": 150,
"n_unanswerable": 150,
"ndcg_at_10": 0.4447119968977239,
"recall_at_1": 0.2391111111111111,
"recall_at_10": 0.5912222222222222,
"recall_at_5": 0.4597777777777778,
"top_hit_from_holder_unanswerable": 0.0
},
"image": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.5708333333333333,
"hit_at_10_wilson_lower": 0.5075830430993526,
"mrr": 0.4262020502645502,
"n_answerable": 240,
"n_unanswerable": 240,
"ndcg_at_10": 0.45449292583858186,
"recall_at_1": 0.35208333333333336,
"recall_at_10": 0.5625,
"recall_at_5": 0.5083333333333333,
"top_hit_from_holder_unanswerable": 0.0
},
"pdf": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.685378590078329,
"hit_at_10_wilson_lower": 0.6516380300993592,
"mrr": 0.518995296116706,
"n_answerable": 766,
"n_unanswerable": 218,
"ndcg_at_10": 0.5587707901607282,
"recall_at_1": 0.44386422976501305,
"recall_at_10": 0.685378590078329,
"recall_at_5": 0.6174934725848564,
"top_hit_from_holder_unanswerable": 0.0
},
"ui_screenshot": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.11875,
"hit_at_10_wilson_lower": 0.08775049817160425,
"mrr": 0.05682539682539682,
"n_answerable": 320,
"n_unanswerable": 320,
"ndcg_at_10": 0.0711722029605294,
"recall_at_1": 0.0375,
"recall_at_10": 0.11875,
"recall_at_5": 0.08125,
"top_hit_from_holder_unanswerable": 0.0
},
"video": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.16279069767441862,
"hit_at_10_wilson_lower": 0.1194411092694439,
"mrr": 0.08660022148394242,
"n_answerable": 215,
"n_unanswerable": 215,
"ndcg_at_10": 0.10455353510193785,
"recall_at_1": 0.05581395348837209,
"recall_at_10": 0.16279069767441862,
"recall_at_5": 0.12093023255813953,
"top_hit_from_holder_unanswerable": 0.0
}
},
"closed_items": 27072,
"items": 403859,
"latency_seconds": {
"max": 0.2064202700057649,
"mean": 0.08776056289499023,
"p50": 0.08638653549860464,
"p95": 0.10662773229960291,
"queries": 5308
},
"overall": {
"abstention_rate_answerable": 0.0,
"abstention_rate_unanswerable": 0.0,
"hit_at_10": 0.43306010928961747,
"hit_at_10_wilson_lower": 0.41521181569494575,
"mrr": 0.29650609332986383,
"n_answerable": 2928,
"n_unanswerable": 2380,
"ndcg_at_10": 0.32645785820497475,
"recall_at_1": 0.23168260473588342,
"recall_at_10": 0.42851206739526415,
"recall_at_5": 0.3708902550091075,
"top_hit_from_holder_unanswerable": 0.0
},
"store": "zep_temporal [whole corpus]"
}
]
}
bench/run_external.py
+26 -1
View File
@@ -12,19 +12,21 @@ from __future__ import annotations
import argparse
import json
import statistics
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from bench.counterpart import (
Counterpart,
SourceMeta,
build_counterparts,
build_items,
load_item_atoms,
load_source_index,
)
from bench.ingest_baseline import RUNS, database_path
from bench.metrics import aggregate, score_query
from bench.metrics import aggregate, chance_recall_at_k, score_query
from bench.run import Query, git_commit, holder_index, load_queries
from bench.sets_common import ROOT
from bench.systems import machine_facts
@@ -37,6 +39,22 @@ SCOPES = (HOLDER_SCOPE, WHOLE_CORPUS_SCOPE)
PROGRESS_EVERY = 500
def chance_rows(index: dict[str, SourceMeta], queries: list[Query]) -> dict[str, Any]:
"""What random guessing scores in each candidate set, so a scoped score is not read as skill."""
per_holder: dict[str, int] = {}
for meta in index.values():
per_holder[meta.holder] = per_holder.get(meta.holder, 0) + 1
sizes = [per_holder[query.holder] for query in queries if query.holder in per_holder]
if not sizes:
raise SystemExit("no query holder appears in the corpus index")
return {
"recall_at_10": chance_recall_at_k(1, len(index), TOP_K),
"recall_at_10_holder_scope": sum(chance_recall_at_k(1, size, TOP_K) for size in sizes) / len(sizes),
"median_holder_sources": float(statistics.median(sizes)),
"note": "expected recall@10 of a random ranking inside the candidate set each row is allowed to see",
}
def score_store(store: Counterpart, queries: list[Query], holder_of: dict[str, str], scope: str) -> dict[str, Any]:
"""Rank every query against one counterpart store and aggregate the same metrics the arms use.
@@ -88,6 +106,12 @@ def write_markdown(report: dict[str, Any], path: Path) -> None:
"- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a "
"generator model, which this round does not have"
),
(
f"- chance: random guessing scores recall@10 {report['chance']['recall_at_10']:.5f} over the whole "
f"corpus and {report['chance']['recall_at_10_holder_scope']:.4f} inside one holder's memory "
f"(median {report['chance']['median_holder_sources']:.0f} sources per holder); read each row against "
"its own floor"
),
"",
(
"| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | "
@@ -166,6 +190,7 @@ def main() -> int:
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
"machine": machine_facts(),
},
"chance": chance_rows(index, queries),
"stores": [score_store(store, queries, holder_of, scope) for store in stores for scope in SCOPES],
}
REPORTS.mkdir(parents=True, exist_ok=True)