Give memories a working tier, a long-term tier, and a record of what disagrees

A memory used to have one state: present. A session could not be reloaded
without searching, nothing separated what a person said today from what the
store ingested a year ago, and two statements that disagree about a number
were both handed to the reader as if the store believed both.

Schema v10. Atoms gain session_id, importance, recall_count, last_recalled_at
and archived_at; two tables arrive beside them — consolidations (one watermark
per holder+scenario lane) and conflicts (a pair, the term they disagree on,
both values, and whether anyone has settled it). v_atoms_current now also
hides archived rows, and apply() drops the view before rebuilding it because
CREATE VIEW IF NOT EXISTS would keep the old body and ignore the archive.

Working memory is a read, not a store: `working <holder> <session>` returns
what that session recorded, in order, with no search involved. Consolidation
rolls a lane's new memories into one standing summary and moves the watermark
to the newest memory actually rolled up — not to the clock of the pass, which
made a second pass count the same memories twice. The archive rule is stated
in the command that runs it (recorded before N days, never returned by a
search, importance at or under a floor) rather than hidden in a decay formula
nobody measured; --dry-run names the victims first.

A conflict is recorded when a new memory repeats a label with a different
number in the same holder's same scenario. Retrieval drops the older side
while the conflict is open and both sides answer again once it is settled:
settling is not superseding, and a word-only disagreement is left alone
because a reworded sentence would fire on every paraphrase. Numbers are keyed
to both the memory's subject and the nearest preceding label, measured: the
two thermostat sentences share only their subject, so a nearest-word-only rule
found nothing.

Search now counts the memories it handed back, which is the only use estimate
the store has, and the archive rule reads it. The dense index applies the same
currency rules as the view, so an archived memory cannot stay findable through
one leg while the other hides it.

Two reads over HTTP (working, conflicts) and two tools over MCP
(working_set, memory_conflicts); curation stays with the operator because an
agent should not decide what its own past is worth. remember and supersede
carry a session id; importance does not cross MCP, because recall_count is
observed and importance is stated.

Gates: 366 passed; ruff clean; mypy 24 errors in 9 files, the recorded 9cb46c1
baseline, net 0; pyright 0 on the touched files; go build/vet/test and gofmt
clean; mcp_write_smoke PASS.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-09 07:13:11 -04:00
1 parent 4c6af87fe0
commit dc9cfb5a30
30 files changed
+1814 -229

No files matched your search

+6
View File
@@ -16,6 +16,7 @@ from .adapters import plan as adapter_plan
from .api_auth import require_write
from .api_cartridges import register as register_cartridge_routes
from .api_holders import register as register_holder_routes
from .api_memory import register as register_memory_routes
from .browse import browse, picture
from .cache_fabric import CacheFabric
from .cartridge import CartridgeService
@@ -219,6 +220,8 @@ def create_app() -> FastAPI:
event_time_precision=request.event_time_precision,
event_time_source=request.event_time_source,
retention_until=request.retention_until,
session_id=request.session_id,
importance=request.importance,
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
@@ -238,6 +241,8 @@ def create_app() -> FastAPI:
event_time=request.event_time,
event_time_precision=request.event_time_precision,
event_time_source=request.event_time_source,
session_id=request.session_id,
importance=request.importance,
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
@@ -257,6 +262,7 @@ def create_app() -> FastAPI:
register_cartridge_routes(app, settings_dep)
register_holder_routes(app, settings_dep)
register_memory_routes(app, settings_dep)
@app.get("/api/adapters", response_model=list[AdapterStatus])
def adapters(settings: Annotated[Settings, Depends(settings_dep)]) -> list[AdapterStatus]:
+48
View File
@@ -0,0 +1,48 @@
"""The memory tiers over HTTP: what a session holds, and what the store disagrees about.
Only the two reads an agent needs are exposed. Curation — rolling a lane up, archiving what
nobody uses, settling a disagreement — is the holder's decision, so it lives in the CLI and the
service, not in a route with no caller.
No `from __future__ import annotations` here: FastAPI reads these route signatures, and string
annotations would be resolved against module globals, where the injected settings_dep exists
only as a parameter of register().
"""
from collections.abc import Callable
from typing import Annotated
from fastapi import Depends, FastAPI, HTTPException
from .config import Settings
from .db import Database
from .memory import MemoryService
from .schemas import ConflictRecord, EvidenceAtom
def register(app: FastAPI, settings_dep: Callable[[], Settings]) -> None:
"""Register the tier reads. The holder is named by the caller because the question is about
somebody's memory, not about the server's own identity."""
def service(settings: Settings) -> MemoryService:
return MemoryService(settings, Database(settings))
@app.get("/api/memory/working", response_model=list[EvidenceAtom])
def working_set(
holder: str,
session: str,
settings: Annotated[Settings, Depends(settings_dep)],
) -> list[EvidenceAtom]:
"""What one session recorded, in the order it recorded it, with no search involved."""
try:
return service(settings).working_set(holder, session)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
@app.get("/api/memory/conflicts", response_model=list[ConflictRecord])
def conflicts(
settings: Annotated[Settings, Depends(settings_dep)],
holder: str | None = None,
) -> list[ConflictRecord]:
"""Every disagreement the store noticed and nobody has settled."""
return service(settings).conflicts(holder)
+6 -69
View File
@@ -1,6 +1,5 @@
from __future__ import annotations
from datetime import datetime
from pathlib import Path
import typer
@@ -17,15 +16,15 @@ from .benchmarks import (
)
from .cartridge import CartridgeService
from .cartridge_assets import CartridgeAssetService
from .cli_common import json_dump
from .cli_common import json_dump, moment
from .cli_maintenance import register as register_maintenance_commands
from .cli_memory import register as register_memory_commands
from .cli_senses import register as register_sense_commands
from .cli_senses import senses_report
from .cli_transactive import register as register_transactive_commands
from .config import get_settings
from .db import Database
from .ingestion import IngestionService
from .memory import MemoryService
from .profiles import runtime_profile_report
from .proofs import HashHopEvaluator, PRDProofSuite
from .proofs.contradictions import CORPUS_NAME, build_contradiction_scenarios, write_corpus
@@ -33,7 +32,7 @@ from .proofs.table import render_table
from .research_modules import ResearchModuleService
from .retrieval import RetrievalService
from .runtime import QwenRuntime
from .schemas import MemoryOrigin, SearchFilters, TimePrecision, TrainingAdapterKind
from .schemas import SearchFilters, TrainingAdapterKind
from .training import TrainingService
app = typer.Typer(no_args_is_help=True)
@@ -42,6 +41,7 @@ app = typer.Typer(no_args_is_help=True)
register_transactive_commands(app, lambda: get_settings())
register_sense_commands(app, lambda: get_settings())
register_maintenance_commands(app, lambda: get_settings())
register_memory_commands(app, lambda: get_settings())
@app.command()
@@ -83,15 +83,6 @@ def ingest(path: Path, cartridge_name: str | None = None) -> None:
typer.echo(result.model_dump_json(indent=2))
def _moment(value: str, option: str) -> datetime | None:
if not value:
return None
try:
return datetime.fromisoformat(value)
except ValueError as error:
raise ValueError(f"{option} {value!r} is not an ISO timestamp: {error}") from error
@app.command()
def search(
query: str,
@@ -114,8 +105,8 @@ def search(
holder=holder or None,
scenario=scenario or None,
source_id=source_id or None,
since=_moment(since, "--since"),
until=_moment(until, "--until"),
since=moment(since, "--since"),
until=moment(until, "--until"),
min_trust=None if min_trust < 0 else min_trust,
),
cursor=cursor or None,
@@ -124,60 +115,6 @@ def search(
typer.echo(result.model_dump_json(indent=2))
@app.command()
def remember(
text: str,
holder: str = typer.Option("", "--holder", help="who owns this memory"),
origin: MemoryOrigin = MemoryOrigin.self_,
scenario: str = "",
event_time: str = typer.Option("", "--event-time", help="when it was true, ISO with offset"),
precision: TimePrecision = TimePrecision.unknown,
retention_until: str = typer.Option("", "--retention-until"),
) -> None:
settings = get_settings()
atom = MemoryService(settings).remember(
text,
holder_id=holder or settings.default_holder,
origin=origin,
scenario=scenario,
event_time=_moment(event_time, "--event-time"),
event_time_precision=precision,
retention_until=_moment(retention_until, "--retention-until"),
)
typer.echo(atom.model_dump_json(indent=2))
@app.command()
def supersede(
atom_id: str,
text: str,
holder: str = typer.Option("", "--holder"),
scenario: str = "",
event_time: str = typer.Option("", "--event-time"),
precision: TimePrecision = TimePrecision.unknown,
) -> None:
settings = get_settings()
atom = MemoryService(settings).supersede(
atom_id,
text=text,
holder_id=holder or settings.default_holder,
scenario=scenario,
event_time=_moment(event_time, "--event-time"),
event_time_precision=precision,
)
typer.echo(atom.model_dump_json(indent=2))
@app.command()
def forget(atom_id: str = "", source_id: str = "", reason: str = "") -> None:
report = MemoryService(get_settings()).forget(
atom_id=atom_id or None,
source_id=source_id or None,
reason=reason,
)
typer.echo(report.model_dump_json(indent=2))
@app.command("list-cartridges")
def list_cartridges() -> None:
for manifest in CartridgeService(get_settings()).list_manifests():
+11
View File
@@ -3,7 +3,18 @@
from __future__ import annotations
import json
from datetime import datetime
def json_dump(value: object) -> str:
return json.dumps(value, ensure_ascii=False, indent=2)
def moment(value: str, option: str) -> datetime | None:
"""An ISO timestamp from the command line, with the option named in the error."""
if not value:
return None
try:
return datetime.fromisoformat(value)
except ValueError as error:
raise ValueError(f"{option} {value!r} is not an ISO timestamp: {error}") from error
+130
View File
@@ -0,0 +1,130 @@
"""CLI face for memory: write it, correct it, erase it, and walk the tiers it lives in.
Registered onto the main Typer app so the command names stay flat (`ely-eye remember`,
`ely-eye archive`) while the memory surface stays in one file. Every command answers in JSON.
"""
from __future__ import annotations
from collections.abc import Callable
import typer
from .cli_common import json_dump, moment
from .config import Settings
from .memory import MemoryService
from .schemas import MemoryOrigin, TimePrecision
def register(app: typer.Typer, settings: Callable[[], Settings]) -> None:
"""Add the memory commands. The settings thunk is passed so one patch in a test redirects
every command, the same way the transactive and maintenance faces do."""
def writer() -> MemoryService:
return MemoryService(settings())
def principal(holder: str) -> str:
"""An empty --holder means the local principal, the identity the server writes as."""
return holder or settings().default_holder
@app.command()
def remember(
text: str,
holder: str = typer.Option("", "--holder", help="who owns this memory"),
origin: MemoryOrigin = MemoryOrigin.self_,
scenario: str = "",
event_time: str = typer.Option("", "--event-time", help="when it was true, ISO with offset"),
precision: TimePrecision = TimePrecision.unknown,
retention_until: str = typer.Option("", "--retention-until"),
session: str = typer.Option("", "--session", help="the session this memory belongs to"),
importance: float = typer.Option(-1.0, "--importance", help="0..1; below 0 leaves it unset"),
) -> None:
"""Store one statement as a memory. Repeating it as the same holder updates it."""
atom = writer().remember(
text,
holder_id=principal(holder),
origin=origin,
scenario=scenario,
event_time=moment(event_time, "--event-time"),
event_time_precision=precision,
retention_until=moment(retention_until, "--retention-until"),
session_id=session,
importance=None if importance < 0 else importance,
)
typer.echo(atom.model_dump_json(indent=2))
@app.command()
def supersede(
atom_id: str,
text: str,
holder: str = typer.Option("", "--holder"),
scenario: str = "",
event_time: str = typer.Option("", "--event-time"),
precision: TimePrecision = TimePrecision.unknown,
session: str = typer.Option("", "--session"),
) -> None:
"""Close a memory and record this one as its replacement."""
atom = writer().supersede(
atom_id,
text=text,
holder_id=principal(holder),
scenario=scenario,
event_time=moment(event_time, "--event-time"),
event_time_precision=precision,
session_id=session,
)
typer.echo(atom.model_dump_json(indent=2))
@app.command()
def forget(atom_id: str = "", source_id: str = "", reason: str = "") -> None:
"""Delete a memory and everything derived from it: vectors, cache, edges."""
report = writer().forget(atom_id=atom_id or None, source_id=source_id or None, reason=reason)
typer.echo(report.model_dump_json(indent=2))
@app.command()
def working(holder: str, session: str) -> None:
"""What one session recorded, in order, with no search involved."""
atoms = writer().working_set(holder, session)
typer.echo(json_dump([atom.model_dump(mode="json") for atom in atoms]))
@app.command()
def consolidate(scenario: str = "", holder: str = typer.Option("", "--holder")) -> None:
"""Roll a lane's new memories into its standing summary and move the watermark."""
record = writer().consolidate(principal(holder), scenario)
typer.echo("nothing new to roll up" if record is None else record.model_dump_json(indent=2))
@app.command()
def archive(
older_than_days: int = typer.Option(90, "--older-than-days"),
max_importance: float = typer.Option(-1.0, "--max-importance", help="archive at or under this"),
dry_run: bool = typer.Option(False, "--dry-run", help="name the memories without moving them"),
) -> None:
"""Move never-used, never-valued memories out of the current tier."""
report = writer().archive_stale(
older_than_days=older_than_days,
max_importance=None if max_importance < 0 else max_importance,
dry_run=dry_run,
)
typer.echo(report.model_dump_json(indent=2))
@app.command()
def archived(holder: str = typer.Option("", "--holder")) -> None:
"""The long-term tier: what the current view is hiding."""
atoms = writer().archived(holder or None)
typer.echo(json_dump([atom.model_dump(mode="json") for atom in atoms]))
@app.command("memory-restore")
def restore(atom_ids: list[str]) -> None:
"""Bring archived memories back. Named apart from `restore`, which brings the store back."""
typer.echo(json_dump({"restored": writer().restore(atom_ids)}))
@app.command()
def conflicts(holder: str = typer.Option("", "--holder")) -> None:
"""Every disagreement the store noticed and nobody has settled."""
rows = writer().conflicts(holder or None)
typer.echo(json_dump([row.model_dump(mode="json") for row in rows]))
@app.command("resolve-conflicts")
def resolve_conflicts(conflict_ids: list[int]) -> None:
"""Settle disagreements so retrieval stops choosing a side for the reader."""
typer.echo(json_dump({"resolved": writer().resolve_conflicts(conflict_ids)}))
+7 -3
View File
@@ -82,9 +82,9 @@ class Database:
token_equivalent, metadata_json, holder_id, origin, scenario,
event_time, event_time_end, event_time_precision, event_time_source,
valid_from, valid_to, supersedes_json, visibility_json, retention_until,
quarantined
quarantined, session_id, importance, recall_count, last_recalled_at, archived_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(atom_id) DO UPDATE SET
modality=excluded.modality,
source=excluded.source,
@@ -109,7 +109,11 @@ class Database:
supersedes_json=excluded.supersedes_json,
visibility_json=excluded.visibility_json,
retention_until=excluded.retention_until,
quarantined=excluded.quarantined
quarantined=excluded.quarantined,
session_id=excluded.session_id,
importance=excluded.importance
-- recall_count, last_recalled_at and archived_at stay as the store has them: a
-- re-ingested copy of the same atom cannot erase how often it was used.
""",
[atom_params(atom) for atom in atoms],
)
+269
View File
@@ -0,0 +1,269 @@
"""The memory tiers, kept out of the query layer the same way the search page is.
Working memory is what one session recorded. The ledger says what has already been rolled up.
Archived rows sit outside the current view on purpose. Recall counts are what the store observed,
not what a caller claimed, so they are written here and nowhere else.
"""
from __future__ import annotations
import sqlite3
from datetime import datetime
from .db import Database
from .row_codec import from_iso, from_json, row_to_atom, to_json
from .schemas import ConflictRecord, ConsolidationRecord, EvidenceAtom
def _moment(value: str) -> datetime:
"""A ledger timestamp is never null, so a row whose date will not parse is corrupt, not empty."""
moment = from_iso(value)
if moment is None:
raise ValueError(f"stored timestamp {value!r} is unreadable")
return moment
def touch_recalls(database: Database, atom_ids: list[str], at: datetime) -> None:
"""Record that a search returned these memories, so use is a fact the store holds."""
if not atom_ids:
return
with database.connect() as connection:
connection.execute(
"""
UPDATE atoms
SET recall_count = recall_count + 1, last_recalled_at = ?
WHERE atom_id IN (SELECT value FROM json_each(?))
""",
(at.isoformat(), to_json(atom_ids)),
)
def session_atoms(database: Database, holder_id: str, session_id: str) -> list[EvidenceAtom]:
"""What this session recorded and nothing else: the short-term tier, read without searching."""
with database.connect() as connection:
rows = connection.execute(
"SELECT * FROM v_atoms_current WHERE holder_id = ? AND session_id = ? ORDER BY ingested_at, atom_id",
(holder_id, session_id),
).fetchall()
return [row_to_atom(row) for row in rows]
def lane_atoms(
database: Database, holder_id: str, scenario: str, *, after: datetime | None = None
) -> list[EvidenceAtom]:
"""Current memories of one holder and scenario, oldest first, from a watermark onward."""
watermark = "" if after is None else after.isoformat()
with database.connect() as connection:
rows = connection.execute(
"""
SELECT * FROM v_atoms_current
WHERE holder_id = ? AND scenario = ? AND (? = '' OR ingested_at > ?)
ORDER BY ingested_at, atom_id
""",
(holder_id, scenario, watermark, watermark),
).fetchall()
return [row_to_atom(row) for row in rows]
def read_ledger(database: Database, holder_id: str, scenario: str) -> ConsolidationRecord | None:
with database.connect() as connection:
row = connection.execute(
"SELECT * FROM consolidations WHERE holder_id = ? AND scenario = ?", (holder_id, scenario)
).fetchone()
if row is None:
return None
return ConsolidationRecord(
holder_id=row["holder_id"],
scenario=row["scenario"],
watermark_at=_moment(row["watermark_at"]),
rollup_atom_id=row["rollup_atom_id"],
atom_count=int(row["atom_count"]),
token_equivalent=int(row["token_equivalent"]),
consolidated_at=_moment(row["consolidated_at"]),
)
def write_ledger(database: Database, record: ConsolidationRecord) -> None:
with database.connect() as connection:
connection.execute(
"""
INSERT INTO consolidations (
holder_id, scenario, watermark_at, rollup_atom_id, atom_count, token_equivalent, consolidated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(holder_id, scenario) DO UPDATE SET
watermark_at=excluded.watermark_at,
rollup_atom_id=excluded.rollup_atom_id,
atom_count=excluded.atom_count,
token_equivalent=excluded.token_equivalent,
consolidated_at=excluded.consolidated_at
""",
(
record.holder_id,
record.scenario,
record.watermark_at.isoformat(),
record.rollup_atom_id,
record.atom_count,
record.token_equivalent,
record.consolidated_at.isoformat(),
),
)
def record_conflict(
database: Database,
*,
older_atom_id: str,
newer_atom_id: str,
holder_id: str,
scenario: str,
terms: list[str],
older_value: str,
newer_value: str,
detected_at: datetime,
) -> ConflictRecord | None:
"""Note that two memories cannot both be current. A repeat of the same pair is not a second conflict."""
with database.connect() as connection:
connection.execute(
"""
INSERT OR IGNORE INTO conflicts (
older_atom_id, newer_atom_id, holder_id, scenario, terms_json,
older_value, newer_value, detected_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(
older_atom_id,
newer_atom_id,
holder_id,
scenario,
to_json(terms),
older_value,
newer_value,
detected_at.isoformat(),
),
)
row = connection.execute(
"SELECT * FROM conflicts WHERE older_atom_id = ? AND newer_atom_id = ?",
(older_atom_id, newer_atom_id),
).fetchone()
return _row_to_conflict(row)
def open_conflicts(database: Database, holder_id: str | None = None) -> list[ConflictRecord]:
"""Every disagreement nobody has picked a side in, newest detection first."""
with database.connect() as connection:
rows = connection.execute(
"""
SELECT * FROM conflicts
WHERE resolved_at IS NULL AND (? = '' OR holder_id = ?)
ORDER BY detected_at DESC, conflict_id
""",
(holder_id or "", holder_id or ""),
).fetchall()
return [_row_to_conflict(row) for row in rows]
def losing_sides(database: Database, atom_ids: list[str]) -> dict[str, str]:
"""Among these atoms, which lost to a newer one in a conflict nobody resolved."""
if not atom_ids:
return {}
payload = to_json(atom_ids)
with database.connect() as connection:
rows = connection.execute(
"""
SELECT older_atom_id, newer_atom_id FROM conflicts
WHERE resolved_at IS NULL
AND older_atom_id IN (SELECT value FROM json_each(?))
AND newer_atom_id IN (SELECT value FROM json_each(?))
""",
(payload, payload),
).fetchall()
return {str(row["older_atom_id"]): str(row["newer_atom_id"]) for row in rows}
def resolve_conflicts(database: Database, conflict_ids: list[int], at: datetime) -> int:
"""Settle disagreements so retrieval stops suppressing a side on the store's own judgement."""
if not conflict_ids:
return 0
with database.connect() as connection:
cursor = connection.execute(
"""
UPDATE conflicts SET resolved_at = ?
WHERE resolved_at IS NULL AND conflict_id IN (SELECT value FROM json_each(?))
""",
(at.isoformat(), to_json([str(conflict_id) for conflict_id in conflict_ids])),
)
return int(cursor.rowcount)
def archive_atoms(database: Database, atom_ids: list[str], at: datetime) -> int:
"""Move memories out of the current tier. The rows stay; only the view stops showing them."""
if not atom_ids:
return 0
with database.connect() as connection:
cursor = connection.execute(
"""
UPDATE atoms SET archived_at = ?
WHERE archived_at IS NULL AND atom_id IN (SELECT value FROM json_each(?))
""",
(at.isoformat(), to_json(atom_ids)),
)
return int(cursor.rowcount)
def restore_atoms(database: Database, atom_ids: list[str]) -> int:
if not atom_ids:
return 0
with database.connect() as connection:
cursor = connection.execute(
"UPDATE atoms SET archived_at = NULL WHERE atom_id IN (SELECT value FROM json_each(?))",
(to_json(atom_ids),),
)
return int(cursor.rowcount)
def archived_atoms(database: Database, holder_id: str | None = None) -> list[EvidenceAtom]:
"""The long-term tier: archived rows, which the current view hides by design."""
with database.connect() as connection:
rows = connection.execute(
"""
SELECT * FROM atoms
WHERE archived_at IS NOT NULL AND (? = '' OR holder_id = ?)
ORDER BY archived_at, atom_id
""",
(holder_id or "", holder_id or ""),
).fetchall()
return [row_to_atom(row) for row in rows]
def stale_candidates(
database: Database, *, before: datetime, max_importance: float | None
) -> list[str]:
"""Current memories that were ingested before the cutoff, were never returned by a search, and
were never marked more important than the floor; an unset importance is not a reason to keep.
"""
with database.connect() as connection:
rows = connection.execute(
"""
SELECT atom_id FROM v_atoms_current
WHERE ingested_at < ? AND recall_count = 0
AND (? IS NULL OR importance IS NULL OR importance <= ?)
ORDER BY ingested_at, atom_id
""",
(before.isoformat(), max_importance, max_importance),
).fetchall()
return [str(row["atom_id"]) for row in rows]
def _row_to_conflict(row: sqlite3.Row) -> ConflictRecord:
return ConflictRecord(
conflict_id=int(row["conflict_id"]),
older_atom_id=row["older_atom_id"],
newer_atom_id=row["newer_atom_id"],
holder_id=row["holder_id"],
scenario=row["scenario"],
terms=[str(item) for item in from_json(row["terms_json"], [])],
older_value=row["older_value"],
newer_value=row["newer_value"],
detected_at=_moment(row["detected_at"]),
resolved_at=from_iso(row["resolved_at"]),
)
+185 -3
View File
@@ -9,15 +9,18 @@ from __future__ import annotations
import hashlib
import json
from datetime import datetime
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
from . import trust
from . import lifecycle_store, trust
from .cache_fabric import CacheFabric
from .config import Settings, get_settings
from .db import Database
from .schemas import (
ArchiveReport,
ConflictRecord,
ConsolidationRecord,
EvidenceAtom,
ForgetReport,
MemoryOrigin,
@@ -32,9 +35,12 @@ from .storage import token_equivalent
from .vector_index import VectorIndex
from .visual_index import VisualIndex
# Trust scores describe extraction fidelity, not truthfulness: a remembered sentence is stored verbatim, so origin carries the epistemic weight.
# A remembered sentence is stored verbatim, so trust scores carry no weight here; origin does.
MEMORY_TRUST = TrustScores(parser=1.0, ocr=0.0, model=0.0)
# The rollup atom's id suffix: one standing summary per holder and scenario, replaced in place.
ROLLUP_SUFFIX = ":rollup"
class MemoryService:
def __init__(
@@ -63,6 +69,8 @@ class MemoryService:
supersedes: list[str] | None = None,
valid_from: datetime | None = None,
retention_until: datetime | None = None,
session_id: str = "",
importance: float | None = None,
) -> EvidenceAtom:
"""Store one statement as a memory owned by a holder.
@@ -116,12 +124,44 @@ class MemoryService:
valid_from=valid_from,
visibility=visibility or Visibility(),
retention_until=retention_until,
session_id=session_id,
importance=importance,
)
trust.stamp([atom], origin=origin)
self.db.upsert_atoms([atom])
self.vectors.sync_atoms([atom.atom_id])
self._note_conflicts(atom)
return atom
def _note_conflicts(self, atom: EvidenceAtom) -> None:
"""A number that moves inside one holder's scenario is a disagreement, not a second fact.
Only a memory that replaces nothing can start a conflict: a supersede has already
picked a side, and recording it as unresolved would contradict that.
"""
if atom.supersedes:
return
stated = trust.numeric_values(atom.text)
if not stated:
return
for earlier in lifecycle_store.lane_atoms(self.db, atom.holder_id, atom.scenario):
if earlier.atom_id == atom.atom_id:
continue
term = trust.conflicting_number(trust.numeric_values(earlier.text), stated)
if term is None:
continue
lifecycle_store.record_conflict(
self.db,
older_atom_id=earlier.atom_id,
newer_atom_id=atom.atom_id,
holder_id=atom.holder_id,
scenario=atom.scenario,
terms=[term],
older_value=trust.numeric_values(earlier.text)[term],
newer_value=stated[term],
detected_at=utc_now(),
)
def supersede(
self,
atom_id: str,
@@ -133,6 +173,8 @@ class MemoryService:
event_time: datetime | None = None,
event_time_precision: TimePrecision = TimePrecision.unknown,
event_time_source: str = "",
session_id: str = "",
importance: float | None = None,
) -> EvidenceAtom:
"""Close an old memory and record the new one as its replacement.
@@ -159,6 +201,8 @@ class MemoryService:
event_time_source=event_time_source,
supersedes=[atom_id],
valid_from=moment,
session_id=session_id,
importance=importance,
)
self.db.close_atoms([atom_id], moment)
self.vectors.sync_atoms([atom_id])
@@ -206,6 +250,144 @@ class MemoryService:
reason=reason,
)
def working_set(self, holder_id: str, session_id: str) -> list[EvidenceAtom]:
"""What this session recorded, in the order it recorded it: the short-term tier.
No search is involved. A session asking for its own context gets exactly what it put
down, which is the whole point of a working set rather than a wider top-k.
"""
if not session_id.strip():
raise ValueError("a working set needs a session; without one every memory is working memory")
return lifecycle_store.session_atoms(self.db, holder_id, session_id)
def consolidate(
self, holder_id: str, scenario: str = "", *, at: datetime | None = None
) -> ConsolidationRecord | None:
"""Roll a lane's new memories into one standing summary and move the watermark.
The summary replaces its own previous version under a fixed id, so a lane holds one
rollup instead of a stack of them, and the ledger's watermark is what keeps a memory out
of the next pass. Nothing is deleted: the individual memories stay current and findable,
because a summary that erased the detail could not be questioned about where it came from.
"""
moment = at or utc_now()
ledger = lifecycle_store.read_ledger(self.db, holder_id, scenario)
batch = [
atom
for atom in lifecycle_store.lane_atoms(
self.db, holder_id, scenario, after=ledger.watermark_at if ledger else None
)
if not atom.atom_id.endswith(ROLLUP_SUFFIX)
]
if not batch:
return None
lines: list[str] = []
if ledger:
previous = self.db.get_atoms([ledger.rollup_atom_id])
if previous:
lines.append(previous[0].text)
lines.extend(atom.text for atom in batch)
body = "\n".join(lines)
digest = hashlib.sha256(f"{holder_id}\n{scenario}".encode()).hexdigest()
source_id = f"rollup-{digest[:16]}"
atom_id = f"{source_id}{ROLLUP_SUFFIX}"
self.db.upsert_sources(
[
SourceRecord(
source_id=source_id,
path=f"memory://{holder_id}/{scenario or 'general'}#rollup",
kind="memory",
mime="text/plain",
sha256=hashlib.sha256(body.encode("utf-8")).hexdigest(),
size_bytes=len(body.encode("utf-8")),
created_at=moment,
metadata={"holder_id": holder_id, "rollup": True},
)
]
)
rollup = EvidenceAtom(
atom_id=atom_id,
modality=Modality.text,
source_id=source_id,
source=f"memory://{holder_id}/{scenario or 'general'}#rollup",
ingested_at=moment,
text=body,
trust=MEMORY_TRUST,
token_equivalent=token_equivalent(body),
holder_id=holder_id,
origin=MemoryOrigin.self_,
scenario=scenario,
)
self.db.upsert_atoms([rollup])
self.vectors.sync_atoms([atom_id])
# The watermark is the newest memory actually rolled up, not the clock of the pass: a
# consolidation run with a clock behind the memories would otherwise count them twice.
record = ConsolidationRecord(
holder_id=holder_id,
scenario=scenario,
watermark_at=max(atom.ingested_at for atom in batch),
rollup_atom_id=atom_id,
atom_count=(ledger.atom_count if ledger else 0) + len(batch),
token_equivalent=token_equivalent(body),
consolidated_at=moment,
)
lifecycle_store.write_ledger(self.db, record)
return record
def archive_stale(
self,
*,
older_than_days: int,
max_importance: float | None = None,
at: datetime | None = None,
dry_run: bool = False,
) -> ArchiveReport:
"""Move memories that were never used and never valued out of the current tier.
The rule is stated instead of hidden in a decay formula: a memory goes when it was
recorded before the cutoff, no search ever returned it, and nobody marked it important.
Archived rows can be brought back; deleting them would make a wrong judgement
unrecoverable, and the store's own estimate of what mattered is exactly the kind of thing
that turns out to be wrong.
"""
if older_than_days < 0:
raise ValueError(f"older_than_days must not be negative, got {older_than_days}")
moment = at or utc_now()
cutoff = moment - timedelta(days=older_than_days)
candidates = lifecycle_store.stale_candidates(self.db, before=cutoff, max_importance=max_importance)
moved = 0 if dry_run else lifecycle_store.archive_atoms(self.db, candidates, moment)
if not dry_run and moved != len(candidates):
raise ValueError(f"{len(candidates)} memories were chosen for the archive but {moved} were moved")
floor = "any importance" if max_importance is None else f"importance at or under {max_importance}"
return ArchiveReport(
atom_ids=candidates,
archived_at=moment,
rule=f"recorded before {cutoff.isoformat()}, never returned by a search, {floor}",
dry_run=dry_run,
)
def archived(self, holder_id: str | None = None) -> list[EvidenceAtom]:
"""The long-term tier: what the current view hides, listed rather than guessed at."""
return lifecycle_store.archived_atoms(self.db, holder_id)
def restore(self, atom_ids: list[str]) -> int:
"""Bring archived memories back: the archive is a tier, not a verdict."""
if not atom_ids:
raise ValueError("restore needs atom ids; nothing was named")
return lifecycle_store.restore_atoms(self.db, atom_ids)
def conflicts(self, holder_id: str | None = None) -> list[ConflictRecord]:
"""Every disagreement the store noticed and nobody has settled."""
return lifecycle_store.open_conflicts(self.db, holder_id)
def resolve_conflicts(self, conflict_ids: list[int], *, at: datetime | None = None) -> int:
"""Settle disagreements so retrieval stops dropping a side on the store's own reading."""
if not conflict_ids:
raise ValueError("resolving needs conflict ids; nothing was named")
return lifecycle_store.resolve_conflicts(self.db, conflict_ids, at or utc_now())
def _drop_derived_vectors(self, atom_ids: list[str]) -> tuple[int, bool]:
"""A forgotten memory must stop answering from the derived indexes too.
+50 -2
View File
@@ -16,7 +16,7 @@ import sqlite3
from .storage import token_equivalent, visual_token_equivalent
from .text_index import index_text
SCHEMA_VERSION = 9
SCHEMA_VERSION = 10
ATOM_COLUMNS_V2 = """
atom_id TEXT PRIMARY KEY,
@@ -100,6 +100,37 @@ _BASE_TABLES = """
updated_at TEXT NOT NULL
);
-- What has already been folded into a rollup, one row per holder and scenario. The
-- watermark is what makes consolidation replayable: the next pass starts where the last one
-- stopped, so the same memory is never counted into two rollups.
CREATE TABLE IF NOT EXISTS consolidations (
holder_id TEXT NOT NULL,
scenario TEXT NOT NULL DEFAULT '',
watermark_at TEXT NOT NULL,
rollup_atom_id TEXT NOT NULL REFERENCES atoms(atom_id) ON DELETE CASCADE,
atom_count INTEGER NOT NULL,
token_equivalent INTEGER NOT NULL,
consolidated_at TEXT NOT NULL,
PRIMARY KEY (holder_id, scenario)
);
-- Two memories that cannot both be current. The store records the disagreement even when no
-- person has picked a winner; retrieval reads this table to keep the newer side instead of
-- packing both numbers into the same prompt.
CREATE TABLE IF NOT EXISTS conflicts (
conflict_id INTEGER PRIMARY KEY AUTOINCREMENT,
older_atom_id TEXT NOT NULL REFERENCES atoms(atom_id) ON DELETE CASCADE,
newer_atom_id TEXT NOT NULL REFERENCES atoms(atom_id) ON DELETE CASCADE,
holder_id TEXT NOT NULL,
scenario TEXT NOT NULL DEFAULT '',
terms_json TEXT NOT NULL,
older_value TEXT NOT NULL,
newer_value TEXT NOT NULL,
detected_at TEXT NOT NULL,
resolved_at TEXT,
UNIQUE (older_atom_id, newer_atom_id)
);
CREATE TABLE IF NOT EXISTS cache_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts TEXT NOT NULL,
@@ -151,11 +182,12 @@ tokenize = 'unicode61'
-- per CJK character and per identifier fragment, which is what makes those queries match.
"""
# A memory is usable now when nothing closed it, it had begun by now, and its retention has not lapsed; unknown bounds read as visible, because silence about when a legacy row was true is not evidence that it is false.
# A memory is usable now when nothing closed it, it had begun by now, its retention has not lapsed, and nobody archived it away from the current tier; unknown bounds read as visible, because silence about when a legacy row was true is not evidence that it is false.
VIEWS = """
CREATE VIEW IF NOT EXISTS v_atoms_current AS
SELECT * FROM atoms
WHERE valid_to IS NULL
AND archived_at IS NULL
AND (valid_from IS NULL OR valid_from <= strftime('%Y-%m-%dT%H:%M:%f+00:00', 'now'))
AND (retention_until IS NULL OR retention_until >= strftime('%Y-%m-%dT%H:%M:%f+00:00', 'now'));
@@ -164,6 +196,8 @@ VIEWS = """
CREATE INDEX IF NOT EXISTS idx_atoms_holder ON atoms(holder_id);
CREATE INDEX IF NOT EXISTS idx_atoms_scenario ON atoms(scenario);
CREATE INDEX IF NOT EXISTS idx_atoms_valid_to ON atoms(valid_to);
CREATE INDEX IF NOT EXISTS idx_atoms_session ON atoms(session_id);
CREATE INDEX IF NOT EXISTS idx_atoms_archived ON atoms(archived_at);
"""
# Columns the first version lacked, in the order added; each is nullable or defaulted so an existing row keeps its meaning.
@@ -185,6 +219,16 @@ _V1_TO_V2_COLUMNS = (
# Columns added after the second version, grouped by the schema version that introduced them.
_NEW_ATOM_COLUMNS: tuple[tuple[int, tuple[tuple[str, str], ...]], ...] = (
(9, (("quarantined", "INTEGER NOT NULL DEFAULT 0"),)),
(
10,
(
("session_id", "TEXT NOT NULL DEFAULT ''"),
("importance", "REAL"),
("recall_count", "INTEGER NOT NULL DEFAULT 0"),
("last_recalled_at", "TEXT"),
("archived_at", "TEXT"),
),
),
)
@@ -270,6 +314,10 @@ def apply(connection: sqlite3.Connection) -> None:
for name, definition in columns:
if name not in existing:
connection.execute(f"ALTER TABLE atoms ADD COLUMN {name} {definition}")
if version < 10:
# CREATE VIEW IF NOT EXISTS keeps the old body, and the tenth version changed which rows
# the current tier contains, so an upgraded database has to be told to look again.
connection.execute("DROP VIEW IF EXISTS v_atoms_current")
connection.executescript(VIEWS)
connection.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
+16 -3
View File
@@ -14,11 +14,11 @@ from datetime import datetime
from pathlib import Path
from typing import Any
from . import cartridge_store
from . import cartridge_store, lifecycle_store
from .config import Settings, get_settings
from .db import Database
from .embeddings import EmbeddingService
from .schemas import EvidenceAtom, RetrievalHit, SearchFilters, SearchResult
from .schemas import EvidenceAtom, RetrievalHit, SearchFilters, SearchResult, utc_now
from .store_search import FtsPage, search_atoms
from .vector_index import VectorIndex
from .visual_index import VisualIndex
@@ -89,7 +89,8 @@ class RetrievalService:
rankings.append(list(visual))
for atom_id, cosine in visual.items():
scores[atom_id]["visual_score"] = cosine
hits = self._rank(rankings, scores, limit)
hits = self._settle_conflicts(self._rank(rankings, scores, limit * CANDIDATE_WINDOW))[:limit]
lifecycle_store.touch_recalls(self.db, [hit.atom.atom_id for hit in hits], utc_now())
more = page.has_more or len(rankings[0]) > limit
return SearchResult(
query=query,
@@ -179,6 +180,18 @@ class RetrievalService:
)
return hits
def _settle_conflicts(self, hits: list[RetrievalHit]) -> list[RetrievalHit]:
"""Drop the older side of a disagreement nobody settled, so one page cannot contradict itself.
Both memories stay in the store and this only filters the page; the conflict row names
which side was dropped and why. Settling is the holder's call, and until it happens the
newer statement is what a reader should be shown.
"""
losers = lifecycle_store.losing_sides(self.db, [hit.atom.atom_id for hit in hits])
if not losers:
return hits
return [hit for hit in hits if hit.atom.atom_id not in losers]
def recall(self, query: str, top_k: int | None = None, *, filters: SearchFilters | None = None) -> list[EvidenceAtom]:
"""The atoms a search would put in front of a model, no scoring attached."""
return [hit.atom for hit in self.search(query, top_k, filters=filters).hits]
+10
View File
@@ -70,6 +70,11 @@ def atom_params(atom: EvidenceAtom) -> tuple[Any, ...]:
to_json(atom.visibility.model_dump()),
atom.retention_until.isoformat() if atom.retention_until else None,
1 if atom.quarantined else 0,
atom.session_id,
atom.importance,
atom.recall_count,
atom.last_recalled_at.isoformat() if atom.last_recalled_at else None,
atom.archived_at.isoformat() if atom.archived_at else None,
)
@@ -102,6 +107,11 @@ def row_to_atom(row: sqlite3.Row) -> EvidenceAtom:
visibility=Visibility(**from_json(row["visibility_json"], {})),
retention_until=from_iso(row["retention_until"]),
quarantined=bool(row["quarantined"]),
session_id=row["session_id"],
importance=None if row["importance"] is None else float(row["importance"]),
recall_count=int(row["recall_count"]),
last_recalled_at=from_iso(row["last_recalled_at"]),
archived_at=from_iso(row["archived_at"]),
)
+50 -1
View File
@@ -109,6 +109,11 @@ class EvidenceAtom(BaseModel):
visibility: Visibility = Field(default_factory=Visibility)
retention_until: datetime | None = None
quarantined: bool = False
session_id: str = ""
importance: float | None = Field(default=None, ge=0.0, le=1.0)
recall_count: int = Field(default=0, ge=0)
last_recalled_at: datetime | None = None
archived_at: datetime | None = None
@model_validator(mode="after")
def _times_are_ordered(self) -> EvidenceAtom:
@@ -120,7 +125,7 @@ class EvidenceAtom(BaseModel):
raise ValueError(f"{self.atom_id}: an atom cannot supersede itself")
return self
@field_validator("ingested_at", "event_time", "event_time_end", "valid_from", "valid_to", "retention_until")
@field_validator("ingested_at", "event_time", "event_time_end", "valid_from", "valid_to", "retention_until", "last_recalled_at", "archived_at")
@classmethod
def _times_are_aware(cls, value: datetime | None) -> datetime | None:
if value is not None and value.tzinfo is None:
@@ -281,6 +286,48 @@ class ForgetReport(BaseModel):
reason: str
class ConsolidationRecord(BaseModel):
"""One lane of the long-term ledger: everything up to the watermark has been rolled up, and
the next pass starts there so no memory is counted into two rollups.
"""
holder_id: str
scenario: str = ""
watermark_at: datetime
rollup_atom_id: str
atom_count: int = Field(ge=0)
token_equivalent: int = Field(ge=0)
consolidated_at: datetime
class ConflictRecord(BaseModel):
"""Two memories of one holder that cannot both be current, with the value each side states."""
conflict_id: int
older_atom_id: str
newer_atom_id: str
holder_id: str
scenario: str = ""
terms: list[str] = Field(default_factory=list)
older_value: str
newer_value: str
detected_at: datetime
resolved_at: datetime | None = None
class ArchiveReport(BaseModel):
"""Which memories left the current tier, and the rule that moved them.
`archived_at` is the moment the rule stamps; on a dry run it is the moment it would
stamp, and `atom_ids` is the list the sentence would fall on.
"""
atom_ids: list[str]
archived_at: datetime
rule: str
dry_run: bool = False
class RememberRequest(BaseModel):
"""A memory as a caller states it. Holder, origin and ingestion time are stamped by the
server, so they are absent here on purpose: a client cannot claim to be somebody else.
@@ -292,6 +339,8 @@ class RememberRequest(BaseModel):
event_time_precision: TimePrecision = TimePrecision.unknown
event_time_source: str = ""
retention_until: datetime | None = None
session_id: str = ""
importance: float | None = Field(default=None, ge=0.0, le=1.0)
class SupersedeRequest(RememberRequest):
+70
View File
@@ -8,6 +8,7 @@ second, non-quarantined source says it too.
from __future__ import annotations
import re
from collections.abc import Iterable
from typing import TYPE_CHECKING
from unicodedata import normalize
@@ -192,3 +193,72 @@ def _spread(pairs: list[str]) -> list[str]:
if len(pairs) <= CORROBORATION_TERMS:
return pairs
return [pairs[0], pairs[len(pairs) // 2], pairs[-1]]
def numeric_values(text: str) -> dict[str, str]:
"""Every number a memory states, keyed by what it measures, so a moving value can be named.
A number is attributed twice: to the subject the memory leads with, and to the content word
standing beside it. 「恒温器放在书房,设定 22 度」 and 「恒温器调到 24 度」 phrase the setting
differently but lead with the same thing, and it is that shared label which makes the pair one
disagreement about one quantity rather than two unrelated counts.
"""
try:
tokens = fts_terms(text)
except ValueError:
return {}
subject = _label(tokens[0], first=True) if tokens else ""
values: dict[str, str] = {}
beside = ""
for token in tokens:
stated = _NUMBER.findall(token)
if not stated:
beside = _label(token) or beside
continue
for value in stated:
if subject:
values.setdefault(subject, value)
if beside:
values.setdefault(beside, value)
return values
def conflicting_number(older: dict[str, str], newer: dict[str, str]) -> str | None:
"""The one quantity both memories state a different value for, or None if they do not disagree.
Only a shared label counts: two memories that each mention a number are not in conflict until
they name the same thing. The label is reported, not the verdict — which side is current is a
decision for the holder, and retrieval only drops the older side while the disagreement stands.
"""
for label in sorted(set(older) & set(newer)):
if _as_float(older[label]) != _as_float(newer[label]):
return label
return None
def _as_float(value: str) -> float:
# The regex only accepts numbers, so a value that will not parse means the pattern changed.
return float(value)
def _label(token: str, *, first: bool = False) -> str:
"""The content word a number hangs on: a CJK run's two-character tail, or a plain Latin word.
The subject of a memory is read from the head of its first run instead, because two memories
about the same thing usually phrase the rest differently: 「恒温器放在书房,设定 22 度」 and
「恒温器调到 24 度」 share only their subject, and that is the label which makes them disagree.
"""
letters = "".join(char for char in token if is_cjk(char))
if letters:
pairs = cjk_pairs(letters)
if not pairs:
return ""
return pairs[0] if first else pairs[-1]
word = token.strip(_PUNCTUATION).lower()
if not word or word in STOPWORDS or any(char.isdigit() for char in word):
return ""
return word
_PUNCTUATION = " \t\n.,;:!?。,;:!?、"
_NUMBER = re.compile(r"[-+]?\d+(?:\.\d+)?")
+1
View File
@@ -231,6 +231,7 @@ class VectorIndex:
"""
SELECT atom_id FROM atoms
WHERE atom_id IN (SELECT value FROM json_each(?))
AND archived_at IS NULL
AND (valid_to IS NULL OR valid_to > strftime('%Y-%m-%dT%H:%M:%f+00:00', 'now'))
AND (valid_from IS NULL OR valid_from <= strftime('%Y-%m-%dT%H:%M:%f+00:00', 'now'))
AND (retention_until IS NULL OR retention_until >= strftime('%Y-%m-%dT%H:%M:%f+00:00', 'now'))