0 Fork 0

Files

T

ZacharyZhang-NY 181f6bf57e Give every memory a holder, an origin, a time and a scenario

An EvidenceAtom could say what was remembered and nothing else: not whose memory it
is, not where it came from, not when it was true, not in what situation it may be
used. The baseline measured the cost of that: the same ranker scores recall@10 0.412
over the whole corpus and 0.866 inside one holder's memory, so the missing holder
column is worth more than any ranking change in the ablation.

- schemas.py: MemoryOrigin, TimePrecision, Visibility; EvidenceAtom gains holder_id,
  origin, scenario, event_time / event_time_end / event_time_precision /
  event_time_source, valid_from / valid_to, supersedes, visibility, retention_until.
  `time` is renamed `ingested_at` — ingestion time gets one name, not two.
  A validator refuses valid_to before valid_from, event_time_end before event_time,
  and an atom that supersedes itself.
- migrations.py: schema v2 with PRAGMA user_version. A first-version database is
  upgraded in place (ALTER + RENAME COLUMN), rows intact, and a database newer than
  this build refuses to open instead of being silently downgraded.
- v_atoms_current hides memories that are closed, lapsed or not yet begun; unknown
  bounds read as visible. Time travel is db.atoms_as_of(moment, holder) because a
  SQLite view cannot take a parameter.
- evaluation_schemas.py: the proof and benchmark report shapes move out of schemas.py
  to keep it under the file-size limit; importers follow.
- backend/tests/test_memory_schema.py: 8 tests, including the v1 -> v2 upgrade and the
  validity timeline.

Verification of record: pytest backend/tests bench/tests -q -> 70 passed; ruff check on
every touched file -> all checks passed; mypy (fresh cache) on db.py, migrations.py,
schemas.py, evaluation_schemas.py and backend/tests -> no issues. evals.py carries 40
pre-existing mypy errors at HEAD and 40 after this change (measured on both copies);
its pyright findings are the same debt, and p0-6 rewrites that file. `peft` and
`pyarrow.parquet` "could not be resolved" is the documented pyright artifact: pi-lens
runs it with 0 config files, and both import at runtime.

2026-10-07 14:30:22 -04:00

3.1 KiB

Raw Blame History

BASELINE_EXTERNAL: memory counterparts on the same corpus

generated 2026-10-07T17:26:07+00:00 at commit c4e2982506
5308 queries (2928 answerable) against 50789 sources
machine: 32 cores, load average 1.5 when the report was written
extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have

store	items	closed	recall@10	hit@10	hit@10 Wilson 95% lower	MRR	abstain (unanswerable)	p50 s	p95 s
mem0_flat [holder scope]	403859	0	0.866	0.867	0.855	0.629	0.000	0.084	0.133
mem0_flat [whole corpus]	403859	0	0.412	0.416	0.398	0.283	0.000	0.090	0.174
zep_temporal [holder scope]	403859	27072	0.866	0.867	0.854	0.627	0.058	0.088	0.111
zep_temporal [whole corpus]	403859	27072	0.429	0.433	0.415	0.297	0.000	0.086	0.107

n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}

[holder scope] answers only from the asking holder's memory; [whole corpus] searches every holder's memory, which is what the product's arms are shown. The gap between the two rows is the value of the holder boundary, not of a better ranker.

mem0_flat [holder scope] by modality

modality	n answerable	recall@10	hit@10	Wilson lower	MRR
code	1237	0.952	0.952	0.939	0.695
conversation	150	0.993	1.000	0.975	0.927
image	240	0.646	0.654	0.592	0.505
pdf	766	0.916	0.916	0.895	0.692
ui_screenshot	320	0.622	0.622	0.568	0.330
video	215	0.716	0.716	0.653	0.396

mem0_flat [whole corpus] by modality

modality	n answerable	recall@10	hit@10	Wilson lower	MRR
code	1237	0.349	0.349	0.323	0.216
conversation	150	0.500	0.573	0.493	0.373
image	240	0.525	0.533	0.470	0.362
pdf	766	0.683	0.683	0.649	0.515
ui_screenshot	320	0.075	0.075	0.051	0.033
video	215	0.116	0.116	0.080	0.069

zep_temporal [holder scope] by modality

modality	n answerable	recall@10	hit@10	Wilson lower	MRR
code	1237	0.952	0.952	0.939	0.695
conversation	150	0.978	0.987	0.953	0.905
image	240	0.646	0.654	0.592	0.505
pdf	766	0.916	0.916	0.895	0.692
ui_screenshot	320	0.622	0.622	0.568	0.330
video	215	0.716	0.716	0.653	0.396

zep_temporal [whole corpus] by modality

modality	n answerable	recall@10	hit@10	Wilson lower	MRR
code	1237	0.350	0.350	0.324	0.216
conversation	150	0.591	0.667	0.588	0.429
image	240	0.562	0.571	0.508	0.426
pdf	766	0.685	0.685	0.652	0.519
ui_screenshot	320	0.119	0.119	0.088	0.057
video	215	0.163	0.163	0.119	0.087

3.1 KiB Raw Blame History