ZacharyZhang-NY
181f6bf57e
An EvidenceAtom could say what was remembered and nothing else: not whose memory it is, not where it came from, not when it was true, not in what situation it may be used. The baseline measured the cost of that: the same ranker scores recall@10 0.412 over the whole corpus and 0.866 inside one holder's memory, so the missing holder column is worth more than any ranking change in the ablation. - schemas.py: MemoryOrigin, TimePrecision, Visibility; EvidenceAtom gains holder_id, origin, scenario, event_time / event_time_end / event_time_precision / event_time_source, valid_from / valid_to, supersedes, visibility, retention_until. `time` is renamed `ingested_at` — ingestion time gets one name, not two. A validator refuses valid_to before valid_from, event_time_end before event_time, and an atom that supersedes itself. - migrations.py: schema v2 with PRAGMA user_version. A first-version database is upgraded in place (ALTER + RENAME COLUMN), rows intact, and a database newer than this build refuses to open instead of being silently downgraded. - v_atoms_current hides memories that are closed, lapsed or not yet begun; unknown bounds read as visible. Time travel is db.atoms_as_of(moment, holder) because a SQLite view cannot take a parameter. - evaluation_schemas.py: the proof and benchmark report shapes move out of schemas.py to keep it under the file-size limit; importers follow. - backend/tests/test_memory_schema.py: 8 tests, including the v1 -> v2 upgrade and the validity timeline. Verification of record: pytest backend/tests bench/tests -q -> 70 passed; ruff check on every touched file -> all checks passed; mypy (fresh cache) on db.py, migrations.py, schemas.py, evaluation_schemas.py and backend/tests -> no issues. evals.py carries 40 pre-existing mypy errors at HEAD and 40 after this change (measured on both copies); its pyright findings are the same debt, and p0-6 rewrites that file. `peft` and `pyarrow.parquet` "could not be resolved" is the documented pyright artifact: pi-lens runs it with 0 config files, and both import at runtime.
65 lines
3.1 KiB
Markdown
65 lines
3.1 KiB
Markdown
# BASELINE_EXTERNAL: memory counterparts on the same corpus
|
|
|
|
- generated 2026-10-07T17:26:07+00:00 at commit c4e2982506fa
|
|
- 5308 queries (2928 answerable) against 50789 sources
|
|
- machine: 32 cores, load average 1.5 when the report was written
|
|
- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have
|
|
|
|
| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | abstain (answerable) | abstain (unanswerable) | p50 s | p95 s |
|
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
| mem0_flat [holder scope] | 403859 | 0 | 0.866 | 0.867 | 0.855 | 0.629 | 0.000 | 0.000 | 0.084 | 0.133 |
|
|
| mem0_flat [whole corpus] | 403859 | 0 | 0.412 | 0.416 | 0.398 | 0.283 | 0.000 | 0.000 | 0.090 | 0.174 |
|
|
| zep_temporal [holder scope] | 403859 | 27072 | 0.866 | 0.867 | 0.854 | 0.627 | 0.000 | 0.058 | 0.088 | 0.111 |
|
|
| zep_temporal [whole corpus] | 403859 | 27072 | 0.429 | 0.433 | 0.415 | 0.297 | 0.000 | 0.000 | 0.086 | 0.107 |
|
|
|
|
n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}
|
|
|
|
|
|
`[holder scope]` answers only from the asking holder's memory; `[whole corpus]` searches every
|
|
holder's memory, which is what the product's arms are shown. The gap between the two rows is the
|
|
value of the holder boundary, not of a better ranker.
|
|
|
|
## mem0_flat [holder scope] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
|
|
| conversation | 150 | 0.993 | 1.000 | 0.975 | 0.927 |
|
|
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
|
|
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
|
|
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
|
|
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
|
|
|
|
## mem0_flat [whole corpus] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.349 | 0.349 | 0.323 | 0.216 |
|
|
| conversation | 150 | 0.500 | 0.573 | 0.493 | 0.373 |
|
|
| image | 240 | 0.525 | 0.533 | 0.470 | 0.362 |
|
|
| pdf | 766 | 0.683 | 0.683 | 0.649 | 0.515 |
|
|
| ui_screenshot | 320 | 0.075 | 0.075 | 0.051 | 0.033 |
|
|
| video | 215 | 0.116 | 0.116 | 0.080 | 0.069 |
|
|
|
|
## zep_temporal [holder scope] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
|
|
| conversation | 150 | 0.978 | 0.987 | 0.953 | 0.905 |
|
|
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
|
|
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
|
|
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
|
|
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
|
|
|
|
## zep_temporal [whole corpus] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.350 | 0.350 | 0.324 | 0.216 |
|
|
| conversation | 150 | 0.591 | 0.667 | 0.588 | 0.429 |
|
|
| image | 240 | 0.562 | 0.571 | 0.508 | 0.426 |
|
|
| pdf | 766 | 0.685 | 0.685 | 0.652 | 0.519 |
|
|
| ui_screenshot | 320 | 0.119 | 0.119 | 0.088 | 0.057 |
|
|
| video | 215 | 0.163 | 0.163 | 0.119 | 0.087 |
|