ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give memories a working tier, a long-term tier, and a record of what disagrees

Browse Source 
A memory used to have one state: present. A session could not be reloaded
without searching, nothing separated what a person said today from what the
store ingested a year ago, and two statements that disagree about a number
were both handed to the reader as if the store believed both.

Schema v10. Atoms gain session_id, importance, recall_count, last_recalled_at
and archived_at; two tables arrive beside them — consolidations (one watermark
per holder+scenario lane) and conflicts (a pair, the term they disagree on,
both values, and whether anyone has settled it). v_atoms_current now also
hides archived rows, and apply() drops the view before rebuilding it because
CREATE VIEW IF NOT EXISTS would keep the old body and ignore the archive.

Working memory is a read, not a store: `working <holder> <session>` returns
what that session recorded, in order, with no search involved. Consolidation
rolls a lane's new memories into one standing summary and moves the watermark
to the newest memory actually rolled up — not to the clock of the pass, which
made a second pass count the same memories twice. The archive rule is stated
in the command that runs it (recorded before N days, never returned by a
search, importance at or under a floor) rather than hidden in a decay formula
nobody measured; --dry-run names the victims first.

A conflict is recorded when a new memory repeats a label with a different
number in the same holder's same scenario. Retrieval drops the older side
while the conflict is open and both sides answer again once it is settled:
settling is not superseding, and a word-only disagreement is left alone
because a reworded sentence would fire on every paraphrase. Numbers are keyed
to both the memory's subject and the nearest preceding label, measured: the
two thermostat sentences share only their subject, so a nearest-word-only rule
found nothing.

Search now counts the memories it handed back, which is the only use estimate
the store has, and the archive rule reads it. The dense index applies the same
currency rules as the view, so an archived memory cannot stay findable through
one leg while the other hides it.

Two reads over HTTP (working, conflicts) and two tools over MCP
(working_set, memory_conflicts); curation stays with the operator because an
agent should not decide what its own past is worth. remember and supersede
carry a session id; importance does not cross MCP, because recall_count is
observed and importance is stated.

Gates: 366 passed; ruff clean; mypy 24 errors in 9 files, the recorded 9cb46c1
baseline, net 0; pyright 0 on the touched files; go build/vet/test and gofmt
clean; mcp_write_smoke PASS.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-09 07:13:11 -04:00
1 parent 4c6af87fe0
commit dc9cfb5a30
30 files changed
+1814 -229
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bench/leg_check.py
+24 -138
View File
@@ -21,6 +21,7 @@ import time
from pathlib import Path
from typing import Any
from bench.leg_report import write_markdown
from bench.metrics import aggregate, newest_gold_first, score_query
from bench.query_set import load_queries, scale_probes, source_times
from bench.report import git_commit
@@ -39,6 +40,9 @@ ENCODE_SAMPLE = 200
# The page the product asks for by default; measure() checks it against Settings, so "as shipped" cannot rot into a hypothesis.
SHIPPED_TOP_K = 24
# The modality whose questions carry a question type; measured whole, because 15 probes of 150 cannot separate six categories.
MEMORY_MODALITY = "conversation"
# (label, atoms asked, dense leg, picture leg); the picture leg answers only on a thin page, which is the product's rule.
CONFIGS: list[dict[str, Any]] = [
{"label": "words only", "top_k": 10, "dense": False, "visual": False},
@@ -234,6 +238,10 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
raise SystemExit("the frozen suite carries no answerable query; nothing to score")
print(f"{len(answerable)} answerable probes", flush=True)
times = source_times()
memory = [query for query in load_queries() if query.modality == MEMORY_MODALITY and query.gold]
if not memory:
raise SystemExit(f"the frozen suite carries no answerable {MEMORY_MODALITY} query; the memory tables would be empty")
print(f"{len(memory)} answerable {MEMORY_MODALITY} queries, measured whole beside the probes", flush=True)
vectors = _dense_vectors(home / "data" / "ely_eye.sqlite")
if vectors == 0 and any(config["dense"] for config in CONFIGS):
@@ -280,6 +288,12 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
]
summary = aggregate(scores)
timing = latency(timings)
memory_rankings, _, _ = rank_all(service, ids, memory, int(config["top_k"]))
memory_scores = [
score_query(query.query_id, ranking, set(query.gold), query.holder)
for query, ranking in zip(memory, memory_rankings, strict=True)
]
memory_summary = aggregate(memory_scores)
arms.append(
{
"label": config["label"],
@@ -294,6 +308,14 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
"p95_seconds": timing["p95"],
"median_distinct_sources": statistics.median(distinct),
**_groups(answerable, scores, rankings, times),
"memory": {
"queries": len(memory),
"recall_at_10": memory_summary["recall_at_10"],
"mrr": memory_summary["mrr"],
"hit_at_10": memory_summary["hit_at_10"],
"wilson_lower": memory_summary["hit_at_10_wilson_lower"],
**_groups(memory, memory_scores, memory_rankings, times),
},
}
)
print(
@@ -310,6 +332,8 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
"picture_index": picture,
"encode_rate": rate,
"probes": len(answerable),
"memory_queries": len(memory),
"memory_modality": MEMORY_MODALITY,
"arms": arms,
"machine": machine_facts(),
}
@@ -322,144 +346,6 @@ def _source_ids(database: Any) -> dict[str, str]:
return _corpus_ids(database)
def write_markdown(report: dict[str, Any], path: Path) -> None:
machine = report["machine"]
header = [
"| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |",
"| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
]
rows = []
for arm in report["arms"]:
rows.append(
f"| {arm['label']} | {arm['top_k']} | {arm['recall_at_10']:.3f} | {arm['mrr']:.3f} | "
f"{arm['hit_at_10']:.3f} | {arm['wilson_lower']:.3f} | {arm['p50_seconds'] * 1000:.1f} | "
f"{arm['p95_seconds'] * 1000:.1f} | {arm['median_distinct_sources']} |"
)
modalities = sorted({modality for arm in report["arms"] for modality in arm["by_modality"]})
modality_rows = [
"| leg configuration | " + " | ".join(modalities) + " |",
"| --- |" + " --- |" * len(modalities),
]
for arm in report["arms"]:
modality_rows.append(
f"| {arm['label']} | "
+ " | ".join(f"{arm['by_modality'].get(modality, float('nan')):.3f}" for modality in modalities)
+ " |"
)
counts = report["arms"][0]["modality_counts"]
def category_table(title: str, field: str, sizes_field: str) -> list[str]:
"""A table over the question types the sets label; unlabelled sets stay out of it."""
names = sorted({name for arm in report["arms"] for name in arm[field] if name != "unlabelled"})
if not names:
return []
lines = [
f"## {title}",
"",
"| leg configuration | " + " | ".join(names) + " |",
"| --- |" + " --- |" * len(names),
]
for arm in report["arms"]:
values, sizes = arm[field], arm[sizes_field]
cells = []
for name in names:
if name not in values:
cells.append("—")
elif field == "by_category":
cells.append(f"{values[name]:.3f}")
else:
cells.append(f"{values[name]:.2f} ({sizes[name]})")
lines.append(f"| {arm['label']} | " + " | ".join(cells) + " |")
labelled = report["arms"][0][sizes_field]
lines += [
"",
"_Queries per type: "
+ ", ".join(f"{name} {size}" for name, size in labelled.items() if name != "unlabelled")
+ "._",
]
return lines
path.write_text(
"\n".join(
[
"# LEGS: what each retrieval leg is worth",
"",
f"- commit: `{report['commit']}`, schema v{report['schema_version']}",
(
f"- store: {report['store']} — a copy of {report['source_store']}, which carries "
f"{report['dense_vectors']:,} dense vectors built by bench/dense_scale. The build "
"sits outside the measured latency, and the frozen baseline itself holds no "
"vectors, so a dense arm there would rank an empty index."
),
f"- {report['probes']} answerable queries, each asked as the holder the frozen set assigns it",
*(
[]
if report.get("picture_index") is None
else [
(
f"- picture index: {report['picture_index']['atoms']:,} atoms at "
f"{report['picture_index']['dim']} dimensions, "
f"{report['picture_index']['bytes']:,} bytes, read from "
f"{report['picture_index']['path']}"
),
(
f"- encoder speed: {report['encode_rate']['per_second']:.1f} pictures/s "
f"measured on {report['encode_rate']['pictures']} pictures, so the "
f"{report['picture_index']['atoms']:,} pictures in that index cost about "
f"{report['picture_index']['atoms'] / report['encode_rate']['per_second'] / 60:.1f} "
"minutes to encode"
),
]
),
(
f"- machine: {machine['system']} {machine['release']}, {machine['cpus']} cores, "
f"{machine['threads_per_query']} threads per query, load {machine['load_average_1_5_15']}"
),
"",
"Recall is scored on the first ten **sources** after atoms are collapsed to their source",
"by first hit, which is how every other arm in this repo is scored.",
"",
*header,
*rows,
"",
"## recall@10 by modality",
"",
*modality_rows,
"",
f"_Query counts per modality: {', '.join(f'{name} {count}' for name, count in counts.items())}._",
"",
*category_table("recall@10 by question type", "by_category", "category_counts"),
"",
*category_table(
"the newest of the gold memories came back first",
"newest_gold_first_by_category",
"newest_gold_first_counts",
),
"",
(
"recall@10 counts any gold memory, so a knowledge-update question can score 1.000 "
"while the ranking still leads with the value the speaker has since replaced. The "
"table above asks the sharper question: when the answer lives in several dated "
"memories, is the newest one the first one returned? Only questions with at least two "
"dated gold memories and at least one returned are counted."
),
"",
(
"The picture leg is not a switch this harness flips per row: the product runs it only "
"when the sparse page comes back thin, so these rows measure the product with the leg "
"available, not a forced always-on fusion."
),
"",
(
"_Latency is wall time per search on the machine named above with the thread budget "
"the benchmark pins; a run on a loaded machine is not comparable._"
),
"",
]
),
encoding="utf-8",
)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--home", type=Path, default=DEFAULT_HOME)
bench/leg_report.py
+150
View File
@@ -0,0 +1,150 @@
"""The LEGS page: one markdown table per question the harness was built to answer."""
from pathlib import Path
from typing import Any
def write_markdown(report: dict[str, Any], path: Path) -> None:
machine = report["machine"]
header = [
"| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | newest first | p50 ms | p95 ms | distinct sources |",
"| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
]
rows = []
for arm in report["arms"]:
newest = arm["newest_gold_first"]
rows.append(
f"| {arm['label']} | {arm['top_k']} | {arm['recall_at_10']:.3f} | {arm['mrr']:.3f} | "
f"{arm['hit_at_10']:.3f} | {arm['wilson_lower']:.3f} | "
f"{'—' if newest is None else f'{newest:.2f}'} | {arm['p50_seconds'] * 1000:.1f} | "
f"{arm['p95_seconds'] * 1000:.1f} | {arm['median_distinct_sources']} |"
)
modalities = sorted({modality for arm in report["arms"] for modality in arm["by_modality"]})
modality_rows = [
"| leg configuration | " + " | ".join(modalities) + " |",
"| --- |" + " --- |" * len(modalities),
]
for arm in report["arms"]:
modality_rows.append(
f"| {arm['label']} | "
+ " | ".join(f"{arm['by_modality'].get(modality, float('nan')):.3f}" for modality in modalities)
+ " |"
)
counts = report["arms"][0]["modality_counts"]
def memory_table(title: str, field: str, sizes_field: str) -> list[str]:
"""A table over the question types the memory set labels; unlabelled sets stay out of it."""
rows = [arm["memory"] for arm in report["arms"]]
names = sorted({name for row in rows for name in row[field] if name != "unlabelled"})
if not names:
return []
lines = [
f"## {title}",
"",
"| leg configuration | " + " | ".join(names) + " |",
"| --- |" + " --- |" * len(names),
]
for arm, row in zip(report["arms"], rows, strict=True):
values, sizes = row[field], row[sizes_field]
cells = []
for name in names:
if name not in values:
cells.append("—")
elif field == "by_category":
cells.append(f"{values[name]:.3f}")
else:
cells.append(f"{values[name]:.2f} ({sizes[name]})")
lines.append(f"| {arm['label']} | " + " | ".join(cells) + " |")
sizes = rows[0][sizes_field]
lines += [
"",
"_Queries per type: "
+ ", ".join(f"{name} {size}" for name, size in sizes.items() if name != "unlabelled")
+ "._",
]
return lines
path.write_text(
"\n".join(
[
"# LEGS: what each retrieval leg is worth",
"",
f"- commit: `{report['commit']}`, schema v{report['schema_version']}",
(
f"- store: {report['store']} — a copy of {report['source_store']}, which carries "
f"{report['dense_vectors']:,} dense vectors built by bench/dense_scale. The build "
"sits outside the measured latency, and the frozen baseline itself holds no "
"vectors, so a dense arm there would rank an empty index."
),
f"- {report['probes']} answerable queries, each asked as the holder the frozen set assigns it",
(
f"- memory subset: every answerable {report['memory_modality']} query "
f"({report['memory_queries']}) measured whole, because a 15-probe sample cannot "
"separate six question types; the two tables further down use that subset, not the sample"
),
*(
[]
if report.get("picture_index") is None
else [
(
f"- picture index: {report['picture_index']['atoms']:,} atoms at "
f"{report['picture_index']['dim']} dimensions, "
f"{report['picture_index']['bytes']:,} bytes, read from "
f"{report['picture_index']['path']}"
),
(
f"- encoder speed: {report['encode_rate']['per_second']:.1f} pictures/s "
f"measured on {report['encode_rate']['pictures']} pictures, so the "
f"{report['picture_index']['atoms']:,} pictures in that index cost about "
f"{report['picture_index']['atoms'] / report['encode_rate']['per_second'] / 60:.1f} "
"minutes to encode"
),
]
),
(
f"- machine: {machine['system']} {machine['release']}, {machine['cpus']} cores, "
f"{machine['threads_per_query']} threads per query, load {machine['load_average_1_5_15']}"
),
"",
"Recall is scored on the first ten **sources** after atoms are collapsed to their source",
"by first hit, which is how every other arm in this repo is scored.",
"",
*header,
*rows,
"",
"## recall@10 by modality",
"",
*modality_rows,
"",
f"_Query counts per modality: {', '.join(f'{name} {count}' for name, count in counts.items())}._",
"",
*memory_table("recall@10 by question type, every memory query measured", "by_category", "category_counts"),
"",
*memory_table(
"the newest of the gold memories came back first",
"newest_gold_first_by_category",
"newest_gold_first_counts",
),
"",
(
"recall@10 counts any gold memory, so a knowledge-update question can score 1.000 "
"while the ranking still leads with the value the speaker has since replaced. The "
"table above asks the sharper question: when the answer lives in several dated "
"memories, is the newest one the first one returned? Only questions with at least two "
"dated gold memories and at least one returned are counted."
),
"",
(
"The picture leg is not a switch this harness flips per row: the product runs it only "
"when the sparse page comes back thin, so these rows measure the product with the leg "
"available, not a forced always-on fusion."
),
"",
(
"_Latency is wall time per search on the machine named above with the thread budget "
"the benchmark pins; a run on a loaded machine is not comparable._"
),
"",
]
),
encoding="utf-8",
)
bench/tests/test_leg_check.py
+21 -7
View File
@@ -5,7 +5,7 @@ from __future__ import annotations
from pathlib import Path
from typing import Any
from bench import leg_check
from bench import leg_check, leg_report
def test_every_leg_appears_switched_both_ways() -> None:
@@ -32,11 +32,19 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
"median_distinct_sources": 4,
"by_modality": {"code": 0.953, "pdf": 0.849},
"modality_counts": {"code": 128, "pdf": 86},
"by_category": {"knowledge-update": 1.0, "temporal-reasoning": 0.947, "unlabelled": 0.84},
"category_counts": {"knowledge-update": 25, "temporal-reasoning": 19, "unlabelled": 256},
"newest_gold_first": 0.44,
"newest_gold_first_by_category": {"knowledge-update": 0.4, "temporal-reasoning": 0.526},
"newest_gold_first_counts": {"knowledge-update": 25, "temporal-reasoning": 19},
"memory": {
"queries": 150,
"recall_at_10": 0.965,
"mrr": 0.875,
"hit_at_10": 0.98,
"wilson_lower": 0.921,
"by_category": {"knowledge-update": 1.0, "temporal-reasoning": 0.947},
"category_counts": {"knowledge-update": 25, "temporal-reasoning": 25},
"newest_gold_first": 0.4,
"newest_gold_first_by_category": {"knowledge-update": 0.4, "temporal-reasoning": 0.526},
"newest_gold_first_counts": {"knowledge-update": 19, "temporal-reasoning": 21},
},
}
report: dict[str, Any] = {
"commit": "abcdef123456",
@@ -47,6 +55,8 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
"picture_index": {"path": "bench/runs/leg_check_index", "atoms": 53668, "dim": 512, "bytes": 109912192},
"encode_rate": {"pictures": 200, "seconds": 10.0, "per_second": 20.0},
"probes": 300,
"memory_queries": 150,
"memory_modality": "conversation",
"arms": [arm],
"machine": {
"system": "Linux",
@@ -57,7 +67,7 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
},
}
out = tmp_path / "LEGS.md"
leg_check.write_markdown(report, out)
leg_report.write_markdown(report, out)
text = out.read_text(encoding="utf-8")
assert "words only" in text
assert "0.832" in text and "63.7" in text, "recall and latency must sit in the same row"
@@ -69,7 +79,11 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
"53,668 atoms at 512 dimensions" in text and "44.7" in text
), "the picture index needs its shape in the report, and its build cost derived from a measured rate"
assert (
"knowledge-update" in text and "1.000" in text and "0.40 (25)" in text
"knowledge-update" in text and "1.000" in text and "0.40 (19)" in text
), "the two question types the contract names need their own recall and their own n"
assert (
"every answerable conversation query (150) measured whole" in text
), "a 15-probe sample cannot separate six question types, and the report must say which set the table used"
assert "| 0.44 |" in text, "the probe table needs its own newest-first column"
assert "unlabelled" not in text, "an unlabelled bucket would hide which set a number came from"
assert "abcdef123456" in text and "load_average" not in text