A memory used to have one state: present. A session could not be reloaded
without searching, nothing separated what a person said today from what the
store ingested a year ago, and two statements that disagree about a number
were both handed to the reader as if the store believed both.
Schema v10. Atoms gain session_id, importance, recall_count, last_recalled_at
and archived_at; two tables arrive beside them — consolidations (one watermark
per holder+scenario lane) and conflicts (a pair, the term they disagree on,
both values, and whether anyone has settled it). v_atoms_current now also
hides archived rows, and apply() drops the view before rebuilding it because
CREATE VIEW IF NOT EXISTS would keep the old body and ignore the archive.
Working memory is a read, not a store: `working <holder> <session>` returns
what that session recorded, in order, with no search involved. Consolidation
rolls a lane's new memories into one standing summary and moves the watermark
to the newest memory actually rolled up — not to the clock of the pass, which
made a second pass count the same memories twice. The archive rule is stated
in the command that runs it (recorded before N days, never returned by a
search, importance at or under a floor) rather than hidden in a decay formula
nobody measured; --dry-run names the victims first.
A conflict is recorded when a new memory repeats a label with a different
number in the same holder's same scenario. Retrieval drops the older side
while the conflict is open and both sides answer again once it is settled:
settling is not superseding, and a word-only disagreement is left alone
because a reworded sentence would fire on every paraphrase. Numbers are keyed
to both the memory's subject and the nearest preceding label, measured: the
two thermostat sentences share only their subject, so a nearest-word-only rule
found nothing.
Search now counts the memories it handed back, which is the only use estimate
the store has, and the archive rule reads it. The dense index applies the same
currency rules as the view, so an archived memory cannot stay findable through
one leg while the other hides it.
Two reads over HTTP (working, conflicts) and two tools over MCP
(working_set, memory_conflicts); curation stays with the operator because an
agent should not decide what its own past is worth. remember and supersede
carry a session id; importance does not cross MCP, because recall_count is
observed and importance is stated.
Gates: 366 passed; ruff clean; mypy 24 errors in 9 files, the recorded 9cb46c1
baseline, net 0; pyright 0 on the touched files; go build/vet/test and gofmt
clean; mcp_write_smoke PASS.
One stride over file order, truncated to the probe count, dropped the
tail of the suite: a 300-probe sample held 128 code, 86 pdf, 36
screenshot, 26 image and 24 video queries and no conversation query at
all, so every report built from it measured five of six modalities.
The budget is now split by suite weight with a floor of one probe per
modality, and a test pins that the frozen sample covers all six.
The set's question type rides along on each query, and a second metric
asks the question recall@10 cannot: when the answer lives in several
dated memories, did the newest come back first? A knowledge-update
question scores 1.000 on recall while the ranking still leads with the
value the speaker has since replaced, and that is the failure the next
task is meant to fix.
Sixteen mutating routes now refuse a caller who cannot present the token the
server started with; queries stay open, because a question is not a write. The
environment variable is ELY_EYE_AUTH_TOKEN, following the repo's env_prefix
convention — the shorter name I reached for first returned 403 where the smoke
expected 401, and the server printed the reason.
Words from a stranger are quarantined on the way in: an HTTP upload is
third_party by definition, and a sentence carrying an instruction-shaped phrase
is quarantined whoever wrote it. A quarantined atom stays findable and stays out
of the compiled context until a second, live, unquarantined source agrees with
its rarest words. Corroboration is computed at read time because it is a
property of the corpus now, not of the moment a memory was written; it is
term-level, and the docstring says so.
forget now erases the source row and the compiled contexts beside the atoms, so
a forgotten memory leaves nothing that can still speak. The semantic cache key
carries a store revision, which fixes a real staleness: a context compiled
before new evidence arrived kept its old quarantine decision forever.
bench.audit --product still pointed at evals.py, which the p1-6 split moved into
backend/src/ely_eye/proofs/. It failed fast, but the anti-self-justification
rule had quietly stopped auditing the product; it now walks the package, and a
test pins that the package exists and holds no re-roll.
Gates: pytest 306 passed; ruff clean; mypy cold cache 24 errors in 9 files, the
recorded baseline, net zero; pyright 0 errors on the 23 backend files and the 2
bench files; go build/vet/test and gofmt clean; npm run build 221.39 kB with
eslint exit 0; bench/filesize_check clean; bench.audit --product clean over 31
harness files; scripts/mcp_write_smoke.sh PASS with the gate live across two
servers and the MCP child.
Both reports were produced by a harness that had just changed, so their
stamps named a commit without that code in it. Re-running them on an idle
machine puts the stamp and the source back in sync.
Recall did not move: all seven leg configurations reproduce to the third
decimal, which is the part of the number anyone should act on. Latency moved
within its usual load band, and the encoder probe came in slower this time,
so the picture index's build cost is now quoted as a band rather than a
single remembered figure.
A review of the p2-1 record found seven places where the prose had drifted
away from the reports it cites. None were product defects; all were record
keeping, and record keeping is what a memory system is judged on.
The quantization row mis-transcribed nine latencies, and the two worst made
dimension truncation look twice as fast as the full scan when the report
measures six percent. The extrapolation table scaled latency by atoms and
bytes by vectors under one heading that named neither switch. The picture
index stats existed only in prose, so the harness now reads the index shape
off the artifact and times the encoder on a sample, which reproduces the
hand-timed build within seven percent. The dense arms named a store that
holds no vectors, so the report names the store the vectors came from and
refuses to run where a dense leg would rank an empty index.
The frozen baseline was migrated in place by the product CLI while the
picture index was built, so its pinned mtime, digest and schema version are
re-pinned and the step that moved it is named. The sqlite_vec import added
two mypy errors; the fix is the missing stub declaration, not an inline
ignore, and the count is back to the recorded baseline of twenty-four.
copy_store() moved into bench/scratch.py: systems.py had reached 512 lines
and bench/filesize_check fails anything over 500. The move is behaviour
identical, so the scale reports keep the commit stamp they were generated
against — the code they describe is complete at that commit.
Every number in CHECKLIST p2-1 and p0-3.7 came out of a scratch script: the
kind of artifact a reader cannot re-run, and the kind that quietly drifts
away from the code it describes. These reports are generated by committed
harnesses and each one stamps the commit that carries the code it measures.
DENSE.md 165,973 atoms / 153,857 vectors: build 1421.0s, search p50 75.6ms
p95 93.0ms, 0.998 agreement with a brute-force scan, one write into
a full index p50 518ms, 1,729 bytes per stored vector.
QUANT.md twelve encodings measured on the same vectors: int8 keeps 0.977 of
the ranking for a quarter of the bytes and no speed, bit is 14x
faster and keeps 0.728, so bit is a coarse filter and nothing more.
TERMS.md the stop-word bound swept, not guessed: 0.05 is the measured peak
(recall@10 0.803 at p50 69.5ms) and also the fastest of the rows
that keep recall.
SCALE.md the product's own search at 165,973 atoms: p50 71.7ms against the
frozen baseline's 20.16s.
LEGS.md what each retrieval leg is worth, per modality, with the page size
the product actually asks for in the table.
The acceptance bar the operator set is recall@10 >= 0.90 at n>=300. It is
not met. The best measured row is 0.858 (dense leg on, page widened to 30,
p50 414ms); the shipped configuration gives 0.848 at p50 72.5ms; the
in-repo counterpart at holder scope gives 0.866. The shortfall is
concentrated in picture golds (video 0.583-0.708, ui_screenshot 0.500-0.583,
image 0.750) while the word-shaped ones already pass (code 0.953, pdf 0.919).
Latency passes by three hundred fold; recall does not, and the gap is four
points of cross-modal retrieval, not index scale.
The product fuses a sparse page, a dense vector leg, and a picture leg that
only speaks when the words come back thin. Two of the three are off by
default, and until now no committed artifact said why.
bench/leg_check.py asks the frozen suite's 300 answerable queries as their
own holder, once per configuration, and prints recall, first-hit quality,
latency and per-modality recall in one table. The picture index and the
CLIP snapshot are 48 minutes of encoding, so they are carried in rather
than rebuilt per row; the harness refuses to run a leg whose snapshot is
missing instead of quietly scoring nothing.
The numbers that came back:
- words only, 10 atoms: recall@10 0.832, MRR 0.683, p50 64ms
- words only, the configured 24: 0.848, 0.688, 69ms
- words only, page 30: 0.855, 0.690, 67ms
- dense leg on: 0.845 / 0.858, MRR down, p50 469ms / 411ms
- picture leg available: 0.815 / 0.825, MRR 0.553 / 0.551, p50 148ms / 164ms
So both defaults stay off, and the acceptance bar (recall@10 >= 0.90) is
not met by the product path yet: 0.858 is the best measured row, the
in-repo counterpart at holder scope is 0.866. The loss concentrates in
picture golds, and the current CLIP leg pays more in dilution than it
returns there.
CHECKLIST p2-1 and p0-3.7 now quote the committed reports (DENSE.md,
QUANT.md, SCALE.md, TERMS.md) instead of scratch scripts.
copy_store lived in dense_scale, so the two harnesses that also needed it reached back into a
measurement module for a plumbing helper. It now sits in systems beside the product path it
protects, and bench.run passes a scratch directory to the live check: the product path migrates
whatever database it opens, and the frozen baseline is cited evidence.
COMMON_TERM_FRACTION = 0.05 is a magic number unless something prints the curve it
came from. bench/term_sweep.py asks the frozen suite as its holders would, once per
candidate bound, and writes recall and latency into the same row of
bench/reports/TERMS.md.
rare_terms now resolves its bound when the call runs instead of when the function was
defined, which is the only way a harness can move it. The default value is unchanged,
so the reports stamped at the parent commit still describe this code.
The dense leg used to be a pair of files next to the database: a full .npy
re-encoded from every atom, re-normalized on every query, and stale the moment
anything was ingested. It is now a vec0 table inside the same SQLite file,
written in the same transaction as the atom it describes, so a memory is
queryable the instant it is remembered.
- vector_index.py maintains the table; the writer inserts, the reader never
rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid,
and it is not on the search path.
- Ranking stays in the engine; every boundary (closed, retired, not yet true,
another holder's private memory, the mounted cartridge) is checked once against
the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on
153,857 vectors; the same check over 200 candidates costs 0.2ms. When the
boundaries reject enough candidates the ask widens once, never twice.
- The Go mirror and the web panel carry the new status fields.
- bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size;
bench/scale_check.py now works on a copy so the frozen baseline stays frozen.
Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and
SCALE.md carry the numbers and the machine they were taken on.
Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test
and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the
parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs",
the same class as the two existing qwen_vl_utils lines, and the repo keeps no
type: ignore anywhere.
A split moves code; it does not get to rewrite what the code promises. The
first pass collapsed three multi-line docstrings into one-liners and the
line counts in the checklist were measured before that happened.
- store_search.py: search_atoms keeps its filter-shape rule and the
as_holder sharing rule, FtsPage keeps why the cursor is an offset
- bench/run.py: the module docstring says what the five arms are again
- CHECKLIST: 116/341/83 lines, 11 call sites over 7 methods, and the
migration ladder now cites 181/186/248/250 instead of the parent's lines
Repo-wide multi-line docstrings are back to 114, the parent's count.
The 500-line rule and the one-line-comment rule were enforced only from
memory, so they drifted. bench/filesize_check.py now checks the budget on
every tracked source file, and it passes.
Split, with behaviour proven unchanged by comparing each moved function body
against HEAD byte for byte (53 training bodies, 21 benchmark bodies, 0
differences):
db.py 503 -> 396 store_search.py 105
bench/run.py 609 -> 351 query_set.py 80, report.py 211
benchmarks.py 619 -> 424 benchmark_scoring.py 262
training.py 1630 -> 94 training_{common,encode,metrics,datasets,lora,retrieval}
TrainingService stays as a thin facade for the CLI; the facade's parameter
lists are pinned against the module functions by inspect, which is what
catches a leaked self.
Deleted the eight top-level functions no file called, after fixing the scan
that had reported 201 false positives by skipping the defining file. The
v1/v2/v3 sweep found nothing to delete and says so: every remaining version
tag is an on-disk format marker, a migration step, a fixture name, or a
third-party model name.
Collapsed 39 multi-line comment blocks (86 lines) to one line each, keeping
the measured facts and error strings that justify the code.
269 tests pass; ruff clean; mypy 24 errors in 9 files against a 24-in-6
baseline, net zero; pyright 4, the same four HEAD measures.
Measured, not guessed: every top-level def in every tracked .py file was
counted against every other tracked file, including markdown, scripts, Go
and TypeScript. Eight names had zero references anywhere:
cartridges_holding_atom and clear_mounts (memory.py reads each cartridge's
own sources.json, and set_mounts already clears by replacing the set),
gradient_checkpointing_enabled, group_hits_by_source, ingested_paths,
is_finite_number, parse_runtime_json (the runtime answers in prose now, and
the JSON path was never wired back), write_sources_json.
Typer commands and pytest cases were excluded from the count because a
framework calls them by name. The same scan now returns zero.
263 tests pass; repo-wide mypy is still 24 errors in 6 files, unchanged.
p1-6's acceptance names bench/filesize_check, and the file did not exist:
the 500-line rule was enforced by memory. It now runs, and it fails on
training.py 1639, benchmarks.py 619, bench/run.py 609 and db.py 503. That
is the honest state of the tree, not a passing gate invented after the work.
The console work passed independent review on round three: the reviewer
re-ran the smoke, then broke the pager three ways and watched each break go
red - a pager that does nothing, a pager that appends, and a Previous button
that stops restoring.
The reviewer's round 2 verified all nine round-1 corrections by re-measuring
them, then failed the commit on the one thing it added that the repo's own bar
forbids: a two-line docstring on _TinyDense. p1-6.4 still lists "comments to one
line" as open work, so a new commit should not widen it. The 470MB rationale
already lives in CHECKLIST.md.
Also cites the CI run for the commit under review (37744547992 @ 20c0416, also
success) instead of stopping at the previous one.
CI failed four tests with "SystemExit: bench/models/paraphrase-multilingual-
MiniLM-L12-v2: model not downloaded". The claim those tests make is that the
joint ablation pass ranks exactly like the per-arm pass and that three arms
needing BM25 score the corpus once. That is a claim about the harness, not
about embedding quality, so it should not require a 470MB download to check.
The tests now inject a deterministic 384-dimensional stand-in dense model and
a stand-in cross-encoder at the seam both paths share. The real models still
carry every measurement in bench/reports; nothing measured changed.
Proven by the condition that broke: with both model directories moved out of
the tree, the whole suite passes (253 passed).
The report is regenerated after 10a39c5 so its stamp names the code it
describes. Speed is quoted as a range in the checklist because transcribe
seconds move with machine load (3.26s then 3.12s on this box) while WER,
denominators and word counts do not move at all.
Numbers: 258.18s of audio, WER 0.242 over 533 reference words, 80 audio atoms
and 563 word timestamps read back from the store, 0 out of order and 0 before
their own segment measured on the stored words.
A second review found three numbers left behind when the report was
regenerated with pinned decoding:
- the transcriber docstring claimed 1.7 s for 88.9 s of speech; the report
says 1.39 s, and 1.7 came from the unpinned run. The docstring now names no
number, so it cannot go stale, and it stops calling container seconds speech
with voice activity detection off.
- the checklist still quoted 3.55 s and 61.6x / 73.0x / 97.7x; the frozen
report says 3.26 s and 63.9x / 82.4x / 112.2x.
- the checklist still said 31 speech segments for one mp4; with greedy decoding
that video yields 38, which is where the store's 80 atoms come from
(38 + 39 + 3).
The report also credited the store with timing invariants that were measured on
the model's answer. The harness now recomputes them from the words as stored
(out_of_order_in_store, before_their_segment_in_store) and says so.
Verification: 212 passed, ruff clean, mypy clean on both touched files with a
cold cache, pyright 0 errors.
Generated by .venv/bin/python -m bench.asr_check after 709104e, so the stamp
names the code under measurement instead of its parent. Two passes agreed:
WER 0.242 both times, transcripts byte-identical.
A review of 5289cb4 found the ASR report was not honest arithmetic:
- "audio s" was the last segment's end, not the media length. The real
container durations are 88.93 / 109.18 / 60.07 s, so the report understated
the audio by 48 s and every speed cell was wrong.
- Decoding was unpinned, so a rerun was a new draw: WER moved 0.242 to 0.262
and one video moved 0.278 to 0.583. Greedy single-temperature decoding with
no carry-over now answers the same way twice, and the harness runs the pass
twice to show it.
- The report claimed its WER was an upper bound. It is not a bound in either
direction: a caption that drops words also hides words the model dropped.
The denominators (201 / 296 / 36 reference words) are printed instead.
- "66 speech atoms" was bench-side arithmetic. The store now says 80 audio
atoms carrying 563 word timestamps, read back after a real ingest.
Same silent-skip bug in a second place: a text file over max_text_file_bytes
returned an empty batch with no reason. It raises FileTooLarge with its size,
and a walk records that refusal like any other.
Also fixed from the same review: the Go search tool never published the audio
modality, cli_senses carried a second json_dump, the no-audio-track branch had
a test that fed it markdown, _seconds was an empty wrapper, and the docstring
quoted a speed no report measured.
Gate root cause: bench/tests imports the bench package by name with no root
conftest, so bare pytest could not collect it at all. pythonpath in pyproject
fixes both launch styles; 212 tests pass either way.
Verification: 212 passed (bare pytest and python -m pytest), ruff clean,
mypy clean on the six touched files with a cold cache, repo-wide mypy 24/6
unchanged from HEAD, pyright 0 errors on the touched files, go build/vet/test
and gofmt clean.
An audio file used to produce no memory at all: _ingest_file returned an empty
batch for any suffix it did not know, with no log and no reason. It now raises
UnsupportedFile when you name the file and records a SkippedFile when it was a
directory walk, so a refusal is visible either way.
Speech is measured, not promised. Over 210.5 seconds of real Video-MME audio,
faster-whisper-tiny on CPU int8 transcribed at 51x realtime and reached word
error rate 0.242 against community subtitles nobody here wrote (a caption track
drops words a transcript keeps, so that is an upper bound). 66 speech atoms now
carry 550 word-level timestamps, 0 words out of order and 0 starting before
their own segment. The snapshot is 78.2 MB, under the 500 MB cap, and the report
is bench/reports/ASR.md.
Two library facts were measured rather than assumed. faster-whisper's own audio
loader is broken against PyAV 19 (av.open got an unexpected keyword
'metadata_errors') and it declares av with no version bound, so asr.decode_audio
decodes with PyAV directly and hands the model a float32 array instead of
pinning an old av with no cp314 wheel. And av.Error does not exist on PyAV 19 -
the real base is av.error.FFmpegError - so that is the only thing caught and my
own ValueErrors keep their messages.
Files over the 500-line rule were split rather than excused: ingestion.py (748)
into ingestion_common / ingestion_media / ingestion_repository / ingestion, and
cli.py (544) into cli_senses plus cli_common, following the cli_transactive
register pattern. MediaIngest declares _source_record abstract, so a half-wired
host fails at construction instead of returning None.
Verification: 207 tests pass; ruff clean on the 13 touched files; mypy clean on
them, and repo-wide mypy fell from 29 errors to 24 (5 fixed, none added);
pyright reports 0 errors on the touched files.
A screenshot with no OCR-able words used to be a memory nothing could
reach. It now has a vector, and a question can be asked with it.
- picture space: Xenova/chinese-clip-vit-base-patch16 fp16 ONNX, 361MB,
sha256 0fad978faa89b85f…, CPU-runnable on the onnxruntime the repo
already ships. Text 76ms, image 29ms/image at batch 8. The ONNX contract
was measured, not assumed: it is the fused contrastive graph, so the
unused tower is fed a placeholder row, and the tokenizer must be
BertTokenizer — a CLIP tokenizer emits ids past the embedding table.
- visual_index.py keeps picture vectors in their own family beside the
text ones, keyed by atom id, and refuses to answer when the map and the
array disagree. retrieval.py stays under 500 lines.
- a search can be asked with a picture, and when the words come up thin
the picture ranking joins the fusion — but only then, so a text query
never pays for the encoder.
- visual tokens are counted at 32x32 patches (migration v7): 21,325 image
atoms went from 5,331 to 34,618 tokens, and the library from 29.9M to
84.2M. The 100M capsule fits this 50,278-file corpus and nothing much
larger — that is what the claim actually means.
- the vision-language describer is written and gated: with no snapshot an
atom records "deferred: no vision-language model snapshot"; a broken
model becomes a note on the atom instead of a dead ingest.
- forget now voids the picture index and the copy this process holds, so
a forgotten picture stops answering from RAM.
Measured, six arms, ScreenSpot-v2 (554 icon crops, 408 whole screenshots),
bench/reports/VISUAL.md:
icon pool picture 0.208 words 0.175 chance 0.018
screenshot pool words 0.384 picture 0.215
screenshot to icon 0.108 — CLIP is a whole-image model, not a grounding
model: it cannot point at one small icon inside a screen.
within one screen MRR 0.908 (recall@10 is 1.000 there only because
chance is 1.000 too, so MRR is the only honest number in that row).
ScreenSpot-v2 instructions are all English, so this run does not test the
bilingual claim and the report says so.
Gates: 199 tests, ruff clean on 17 files, mypy clean on 16 (fresh cache),
the repo's own pyright 0 errors on the 12 files touched.
The product's own search answered in p50 20.16s on 165,973 atoms. The SQL
rewrite brought that to 0.68s; this brings it to 57ms and says why the rest was
there.
The cost was not SQL. An OR MATCH scores every row it touches, and a question
carries words the whole library carries: `"parsing"* OR "error"* OR "in"* OR
"the"* OR "pdf"* OR "converter"*` matched 127,145 atoms and took 305ms, while a
CJK bigram phrase query with 32 matches took 1.0ms. Latency tracked match count.
- text_index.rare_terms asks the index which of its own words are noise: count
each term, drop the ones over 5% of the library, keep the rarest few when all
of them are common. No stop-word list; a corpus of PDFs and a corpus of shell
history get different answers. 0.05 is swept, not chosen — on n=173
recall@10 peaks there (0.899 / p50 58.6ms) against 0.893 / 112.9ms at 0.20
and 0.844 / 13.4ms at 0.01.
- The bound is a cost, not a claim about rarity: `cap = max(int(total*fraction)+1, 2)`,
so a word matching one atom in eleven is kept — scoring one candidate is free.
A capped count cannot rank words that all saturate, so those few are counted
exactly.
- Schema v6 adds `term_df`: a word is counted once and the next question pays
nothing. Every atom write clears it, so a stored count cannot outlive the
library it described. Cold p50 73ms, warm 63ms, 1,324 terms cached.
- The denominator is `count(*) FROM atoms` (0.01ms) rather than
`count(*) FROM atom_fts` (37ms cold): FTS rows are written and deleted with
their atoms, so the two counts move together and only one of them is free.
SCALE.md, idle machine (load 1.76, 32 cores, 4 threads per query):
165,973 atoms — 1.66x the contract's 100k — 300 probes, p50 57ms, p95 138ms.
Against the frozen baseline of 20.16s that is 354x. Holder-scoped (n=173):
recall@10 0.870, MRR 0.740, p50 62ms.
Verification: pytest backend/tests bench/tests -q -> 173 passed (5 new
text_index tests: the common word is dropped and the rare one still answers, an
all-common query keeps its rarest words in order, a one-word query is never
filtered, an empty library keeps everything, and a write voids the counts);
ruff clean; mypy fresh cache clean; pyright 0 errors on the touched files.
The frozen baseline claimed the product's top-10 shared 0.0% with the fast
BM25 path. That was a harness bug: gold names a corpus source
(mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content
hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never
met and the right document, ranked first, scored as a miss.
- bench/systems.py: `_corpus_ids` joins store sources through the
`source_map()` the counterpart already uses, and refuses a store whose
sources are not corpus files instead of silently scoring zero.
- bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the
product as the query's own holder, which is how the counterpart is scored.
- bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a
95% lower bound, and probes 300 queries instead of 40 — the SQL path answers
in well under a second, so the old cap was a cost estimate that expired.
- bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen
BASELINE without touching it, stamped with the commit and the store schema
version the pass started on (the live pass migrates the store mid-run, so a
plain stamp would be a lie).
- bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns.
The corpus index states the owner, so holders, scenarios and event times are
stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned.
Numbers, same machine, load 1.0-1.7:
- CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s;
fusion+rerank 0.453 p50 7.000s; graph 0.001.
- Product as its holder: recall@10 0.870, MRR 0.740, n=173.
- Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder
boundary is worth +0.454 with an identical ranker and identical items.
Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy
(fresh cache) clean on every touched file.
The product path pulled every atom into Python and rebuilt a BM25Okapi
index per query: p50 20.16s over 165,973 atoms. It now runs one SQLite
statement with LIMIT/OFFSET.
- text_index.py: one normalizer for both sides of the index. CJK runs are
split per character (unicode61 cannot segment them), camelCase and
separators are split so "search atoms" finds SearchAtoms, latin tokens
get a prefix match. Rows written before this existed hold raw text in
atom_fts, so schema v3 rebuilds the index on upgrade.
- retrieval.py: filters (holder / scenario / source / modality / since /
until / min_trust), a base64 cursor, and reciprocal rank fusion in place
of the 0.55/0.35/0.10 blend whose graph term was always zero.
- An empty result is {status: "no_match"}, not a bare array.
- Go: search_atoms and compile_context rank through POST /api/search. The
second Go tokenizer is deleted; two implementations could drift, which
is what the MCP/HTTP parity requirement forbids.
- db.py passed 500 lines, so the row codec moved to row_codec.py.
- Tooling: installed the repo's own pyright. pyrightconfig.json had no
exclude and swept the downloaded corpus (1171 errors); with excludes the
count is 62, all pre-existing (evals 48, profiles 4, training 4,
ingestion 3, runtime 2, cartridge_assets 1). backend/pyproject.toml
carried a second [tool.pyright] that shadowed the root config; deleted.
- bench/scale_check.py measures the product search at corpus size. The
100k-atom p50 number is not written until it is measured on an idle
machine.
Verification: 112 pytest passed; go build/vet/test clean, gofmt clean;
ruff clean on every touched file; mypy clean on the 14 touched Python
files; repo pyright 62 errors, unchanged per-file ledger.
An EvidenceAtom could say what was remembered and nothing else: not whose memory it
is, not where it came from, not when it was true, not in what situation it may be
used. The baseline measured the cost of that: the same ranker scores recall@10 0.412
over the whole corpus and 0.866 inside one holder's memory, so the missing holder
column is worth more than any ranking change in the ablation.
- schemas.py: MemoryOrigin, TimePrecision, Visibility; EvidenceAtom gains holder_id,
origin, scenario, event_time / event_time_end / event_time_precision /
event_time_source, valid_from / valid_to, supersedes, visibility, retention_until.
`time` is renamed `ingested_at` — ingestion time gets one name, not two.
A validator refuses valid_to before valid_from, event_time_end before event_time,
and an atom that supersedes itself.
- migrations.py: schema v2 with PRAGMA user_version. A first-version database is
upgraded in place (ALTER + RENAME COLUMN), rows intact, and a database newer than
this build refuses to open instead of being silently downgraded.
- v_atoms_current hides memories that are closed, lapsed or not yet begun; unknown
bounds read as visible. Time travel is db.atoms_as_of(moment, holder) because a
SQLite view cannot take a parameter.
- evaluation_schemas.py: the proof and benchmark report shapes move out of schemas.py
to keep it under the file-size limit; importers follow.
- backend/tests/test_memory_schema.py: 8 tests, including the v1 -> v2 upgrade and the
validity timeline.
Verification of record: pytest backend/tests bench/tests -q -> 70 passed; ruff check on
every touched file -> all checks passed; mypy (fresh cache) on db.py, migrations.py,
schemas.py, evaluation_schemas.py and backend/tests -> no issues. evals.py carries 40
pre-existing mypy errors at HEAD and 40 after this change (measured on both copies);
its pyright findings are the same debt, and p0-6 rewrites that file. `peft` and
`pyarrow.parquet` "could not be resolved" is the documented pyright artifact: pi-lens
runs it with 0 config files, and both import at runtime.
rankings.jsonl stores the seconds each query took, so the report inherited whatever
the machine was doing at the time: the frozen baseline's dense latency included a
per-query model load, and the 50,000-source row moved from 1.315s to 0.809s between
two runs of identical code.
- bench/run.py: --remeasure-latency N re-times a stride sample with warm models and
replaces each arm's latency section, recording the sample size and a note; recall
stays a full-pass number. BASELINE.md prints that note under the table.
- bench/tests/test_joint.py: 62 tests total.
Verification: pytest bench/tests -q -> 62 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
Three defects made the external baseline numbers and labels wrong.
- bench/systems.py: _sentence_model rebuilt the model on every query, so dense arm
latency was mostly model loading and the latency-vs-size curve was flat because a
constant dominated it. Models are now cached per process.
- bench/counterpart.py: visible() and search() walked 404k Python objects per query.
Per-item holder/source/time columns are built once and the query path is numpy;
12.5 queries/s instead of 0.7.
- bench/metrics.py: the aggregate key was named wrong_holder_confidence_unanswerable
while it counts top hits that ARE from the asking holder. BASELINE.md read that
backwards. Renamed to top_hit_from_holder_unanswerable and the sentence rewritten.
- bench/run_external.py: the by-modality table read row['answerable'] but aggregate
emits n_answerable, so the writer died after scoring finished. Each store is now
scored twice, [holder scope] and [whole corpus], so the gain from the holder
boundary is separable from the gain from ranking; progress lines every 500 queries.
Verification: pytest bench/tests -q -> 61 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
Latency measured while another job hogs the cores is not comparable: the scale curve
moved from 1.315s to 0.809s at 50,000 sources between two runs of the same code.
- bench/systems.py: machine_facts() reports cpus plus the 1/5/15 minute load average.
- bench/run.py, bench/run_external.py: both reports carry it, and BASELINE.md labels
the latency table a ceiling when the machine was busy.
Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
The frozen baseline showed the unmodified RetrievalService agreeing with an equivalent
BM25 index on 0% of its top ten results. Reproduced by hand: its top ten atoms came
from one document's pages, so a source-level ranking cannot overlap it.
- bench/systems.py: live_product_rankings also counts distinct sources behind the
product's top-k atoms.
- bench/run.py: scale probes no longer filtered by gold (a source prefix keeps almost
no gold, so the 500/2,000/10,000 rows were silently skipped); scale_arm_names reads
only latency dicts, which is what crashed write_markdown on the note field; the
unanswerable share is computed instead of hard-coded.
- bench/tests/test_joint.py: 60 tests total, including the writer regression.
Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the
full ablation at roughly 16 hours and left no way to survive a kill. JointRunner
computes each signal once per query, attributes latency to the components each arm
uses, and checkpoints every query to runs/baseline/rankings.jsonl.
- bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution.
- bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume,
latency summary, resident-memory progress lines.
- bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five
arms; the sparse corpus is scored once even with three arms that need it; cache
fingerprint mismatch and malformed rows fail fast.
- bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which
published numbers are comparable at all.
Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
The merged database names a source by content hash (`11089_4f2a...`) while the
frozen sets name the same file by its corpus id, so every arm scored 0.000 until
load_atoms joined atoms through sources.path. It now returns corpus ids and
refuses an atom it cannot trace to a corpus file.
Also fixes found by running the harness:
- merge rebuilt atom_fts from the deduped atoms table; two attempts on one file
left 288,266 fts rows for 165,973 atoms.
- merge accepts the two honest reasons a corpus file has no product source
(500 byte-identical LongMemEval sessions collapse under content hashing, and
12 photos that ONNX Runtime cannot decode) and reports them.
- --arms used action=extend against a non-empty default, so naming two arms ran
seven.
- the dense cache is keyed by the atom set, so a scale-curve subset cannot be
scored with full-corpus vectors.
- write_markdown crashed on a run that covered only some modalities.
- bench/ingest_baseline.py drives the unmodified product IngestionService over
all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and
memory ceilings are set explicitly: ONNX Runtime defaulted to every core
(95 threads/worker, load 490) and its CPU arena kept freed pages mapped,
which OOM-killed workers.
- bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance
baseline. No model, no product import, so the scorer cannot grade itself.
- bench/systems.py + bench/run.py: the five ablation arms over the merged DB,
latency percentiles, scale curve, and a top-10 equivalence check against the
live RetrievalService.
- bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store
and Zep-style temporal store as the external reference point.
- bench/audit.py: enforces single-attempt scoring, frozen n, a chance row
beside every score, and producer != verifier. It already flags the product's
retry-until-target loop at backend/src/ely_eye/evals.py:2351.
- backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable
(PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.
- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
gold and query, gold observed after as_of, and manifest digest drift
ZacharyZhang-NY