11e31eb7e305624a2556cb7c4e7f55552ac47f65
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ZacharyZhang-NY
|
ff2471d362 |
Give Chinese memories a way to be corroborated
An independent reviewer repeated the reviewer's own counterexample in Chinese and found the corroboration channel was dead for it: fts_terms counts one CJK run as one term, so a Chinese claim could never reach the three words the rule demands, however many independent sources repeated it. A run is now broken into its adjacent two-character pairs and three are taken from across the sentence, so a second source has to agree in three separate places rather than copy one stretch of characters. The pairs belonging to an injection phrase are stripped too, so an instruction cannot vouch for its own characters. Corroboration stays term-level in both languages: a paraphrase that keeps the meaning and changes the wording is not a second witness, and a claim with fewer usable words than the rule demands is never believed. The record was wrong in four places the same review measured: the token rename breakdown summed to 11 while claiming 13 across 6 files (it is 13 across 7), the acceptance cell counted tests that no longer exist, the gate cell quoted the previous round's totals, and a cited line had drifted onto a blank. The section also stops calling the injection work "sanitization": 19 literal phrases plus quarantine plus one prompt sentence is detection, and a reworded directive walks straight past it. |
||
|
ZacharyZhang-NY
|
203b6d04af |
Make corroboration depend on the claim, not on the size of the library
An independent reviewer broke the quarantine rule twice by running it. In a two-atom library, a second source repeating the claim word for word could not vouch for it; 40 unrelated memories could. The old rule handed the choice of words to rare_terms, which drops words common to the corpus, so the corpus size and the attack's own words decided whether a stranger was believed. Corroboration now demands the claim's three longest words, with the instruction phrases stripped out, ANDed by the new text_index.fts_all_expression. Neither the library's size nor the attack can flip it, and both cases are tests. Also from that review: - A question is a read. The DF cache no longer remembers a word the corpus has never used, so an unauthenticated search cannot grow the store one row per novel word. "A question is not a write" was false for /api/search and /api/ask. - The prompt now states that packed evidence is quoted material, not an instruction: once a quarantined sentence is corroborated it does reach the model. - ForgetReport's semantic_cache_cleared reached the frontend as well as Go. - runtime.py: process_vision_info returns a third element only when return_video_kwargs is set, and this call does not set it, so the unpack takes the first two. Two pre-existing pyright errors go, behaviour is unchanged. CHECKLIST corrected against measurement: 13 renamed lines rather than 11, and 10 English / 9 Chinese injection signals rather than 11 / 8. Gates: pytest 312 passed, ruff clean, mypy 24 errors in 9 files (net zero), pyright 0 errors on the five touched files, npm build and eslint clean, scripts/mcp_write_smoke.sh PASS. |
||
|
ZacharyZhang-NY
|
b8f83f422a |
Put the stop-word bound's evidence in the repo, not in a scratch script
COMMON_TERM_FRACTION = 0.05 is a magic number unless something prints the curve it came from. bench/term_sweep.py asks the frozen suite as its holders would, once per candidate bound, and writes recall and latency into the same row of bench/reports/TERMS.md. rare_terms now resolves its bound when the call runs instead of when the function was defined, which is the only way a harness can move it. The default value is unchanged, so the reports stamped at the parent commit still describe this code. |
||
|
ZacharyZhang-NY
|
9cb46c18d3 |
Keep every file inside the rules the repo already wrote down
The 500-line rule and the one-line-comment rule were enforced only from
memory, so they drifted. bench/filesize_check.py now checks the budget on
every tracked source file, and it passes.
Split, with behaviour proven unchanged by comparing each moved function body
against HEAD byte for byte (53 training bodies, 21 benchmark bodies, 0
differences):
db.py 503 -> 396 store_search.py 105
bench/run.py 609 -> 351 query_set.py 80, report.py 211
benchmarks.py 619 -> 424 benchmark_scoring.py 262
training.py 1630 -> 94 training_{common,encode,metrics,datasets,lora,retrieval}
TrainingService stays as a thin facade for the CLI; the facade's parameter
lists are pinned against the module functions by inspect, which is what
catches a leaked self.
Deleted the eight top-level functions no file called, after fixing the scan
that had reported 201 false positives by skipping the defining file. The
v1/v2/v3 sweep found nothing to delete and says so: every remaining version
tag is an on-disk format marker, a migration step, a fixture name, or a
third-party model name.
Collapsed 39 multi-line comment blocks (86 lines) to one line each, keeping
the measured facts and error strings that justify the code.
269 tests pass; ruff clean; mypy 24 errors in 9 files against a 24-in-6
baseline, net zero; pyright 4, the same four HEAD measures.
|
||
|
ZacharyZhang-NY
|
7e1fe649f6 |
Stop paying for words every document carries
The product's own search answered in p50 20.16s on 165,973 atoms. The SQL rewrite brought that to 0.68s; this brings it to 57ms and says why the rest was there. The cost was not SQL. An OR MATCH scores every row it touches, and a question carries words the whole library carries: `"parsing"* OR "error"* OR "in"* OR "the"* OR "pdf"* OR "converter"*` matched 127,145 atoms and took 305ms, while a CJK bigram phrase query with 32 matches took 1.0ms. Latency tracked match count. - text_index.rare_terms asks the index which of its own words are noise: count each term, drop the ones over 5% of the library, keep the rarest few when all of them are common. No stop-word list; a corpus of PDFs and a corpus of shell history get different answers. 0.05 is swept, not chosen — on n=173 recall@10 peaks there (0.899 / p50 58.6ms) against 0.893 / 112.9ms at 0.20 and 0.844 / 13.4ms at 0.01. - The bound is a cost, not a claim about rarity: `cap = max(int(total*fraction)+1, 2)`, so a word matching one atom in eleven is kept — scoring one candidate is free. A capped count cannot rank words that all saturate, so those few are counted exactly. - Schema v6 adds `term_df`: a word is counted once and the next question pays nothing. Every atom write clears it, so a stored count cannot outlive the library it described. Cold p50 73ms, warm 63ms, 1,324 terms cached. - The denominator is `count(*) FROM atoms` (0.01ms) rather than `count(*) FROM atom_fts` (37ms cold): FTS rows are written and deleted with their atoms, so the two counts move together and only one of them is free. SCALE.md, idle machine (load 1.76, 32 cores, 4 threads per query): 165,973 atoms — 1.66x the contract's 100k — 300 probes, p50 57ms, p95 138ms. Against the frozen baseline of 20.16s that is 354x. Holder-scoped (n=173): recall@10 0.870, MRR 0.740, p50 62ms. Verification: pytest backend/tests bench/tests -q -> 173 passed (5 new text_index tests: the common word is dropped and the rare one still answers, an all-common query keeps its rarest words in order, a one-word query is never filtered, an empty library keeps everything, and a write voids the counts); ruff clean; mypy fresh cache clean; pyright 0 errors on the touched files. |
||
|
ZacharyZhang-NY
|
4ab36cd5eb |
Make the proof suite say what it actually ran
The suite had fifteen checks that could not lose. The visual HashHop arena painted a cyan line and a white ring on the answer slot, then asked the model to find the answer by looking; the collector re-rolled puzzles until the running rate crossed its target; the cache check passed if any layer had ever recorded an entry; the repository check passed on a call graph built in memory without touching the store; the research-module check asserted non-empty lists of SHA256-derived placeholder vectors and so could not fail. Each check now answers for one claim and reports a verdict the reader can act on: passed, failed, deferred, or removed. A deferred row names what is missing and the command that would settle it; a removed row says why the old claim was withdrawn. The suite verdict is failed > deferred > passed, so a claim that never ran holds the whole report back from saying "proven". - evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines. - proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a different glob. proofs/table.py renders; latest-proof-suite is the one face and exits 1 on a failed verdict. - The arena paints nothing. The puzzle id is printed on its tile instead, because removing the ink without it removes the question. - HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target. - The contradiction corpus moves from code to contradiction_scenarios.json; the check stays deferred while the corpus is the one this repo authored. - The cache check compiles a nonce question over a real atom twice and reads the miss to hit movement; an empty store fails instead of agreeing by luck. - The repository check requires rows in the store; the research-module check is removed. Two product bugs surfaced on the way: - text_index.fts_query joined Latin terms with a space, which FTS5 reads as AND, so a question longer than its answer retrieved nothing. Terms are ORed and bm25 rank orders the candidates. - profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with an empty drive, producing /mnt//home/... and failing every launcher probe. Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this commit does not touch, measured against the same files at HEAD. |
||
|
ZacharyZhang-NY
|
dfc6cd4c28 |
Give the system a directory: ask who knows, then ask them
A memory system that only answers from its own store is a filing cabinet, not a partner. The transactive rule the project is named for — not knowing is fine when you know who does — needed a holder registry, a directory query, and a referral that never leaks the text it may not read. - schema v5 adds `holders` (person/agent/service, declared topics, contact). A holder with memories exists whether or not anybody registered them; the registry adds reachability, it does not grant existence. - `who_knows` ranks holders by their own memories and is deliberately NOT filtered by visibility: "she knows" is the answer. The answer is filtered. - `ask` searches the asker's visible scope first (hop 0), then walks the directory with a visited set and a hop budget. Three terminal states: answered, referred, no_holder_knows. Nobody knowing is a result, not an error. - Confidence is ordinal, not calibrated: relative strength, coverage, and declared topics. Every hit carries the raw numbers and the basis lines that produced the score, so a ranking can be argued with. - Faces: HTTP (/api/holders, /api/who_knows, /api/ask), CLI (holders, holder-set, holder-rm, who-knows, ask), MCP (who_knows, ask, list_holders). Four bugs found by running the thing, not by reading it: - FTS5 refuses bm25() inside MAX(); the aggregate uses FTS5's own `rank` column. - A CJK question is longer than the sentence that answers it, so the whole-run phrase matched nothing. The query now ORs adjacent bigram phrases; adjacency is kept, equal length is not. Latin path untouched, baseline unaffected. - The hop budget counted the asker as a hop, so max_hops=1 asked nobody. - A second, stricter readability test in _answered() disagreed with the SQL visibility clause and hid a memory the asker was allowed to read. The SQL clause is the only authority now. A method named `list` shadowed the builtin inside its own class and made mypy reject the return annotation; renamed rather than cast around it. Proof: 155 pytest, 12 Go tests, ruff and mypy clean on touched files, pyright ledger unchanged at 62, and scripts/mcp_write_smoke.sh drives two principals on one SQLite file — the husband asks through MCP, gets `referred` with his wife's contact channel, and her private sentence appears nowhere in the reply. |
||
|
ZacharyZhang-NY
|
674c88e84e |
Rank memory search in SQL instead of rebuilding BM25 in Python
The product path pulled every atom into Python and rebuilt a BM25Okapi
index per query: p50 20.16s over 165,973 atoms. It now runs one SQLite
statement with LIMIT/OFFSET.
- text_index.py: one normalizer for both sides of the index. CJK runs are
split per character (unicode61 cannot segment them), camelCase and
separators are split so "search atoms" finds SearchAtoms, latin tokens
get a prefix match. Rows written before this existed hold raw text in
atom_fts, so schema v3 rebuilds the index on upgrade.
- retrieval.py: filters (holder / scenario / source / modality / since /
until / min_trust), a base64 cursor, and reciprocal rank fusion in place
of the 0.55/0.35/0.10 blend whose graph term was always zero.
- An empty result is {status: "no_match"}, not a bare array.
- Go: search_atoms and compile_context rank through POST /api/search. The
second Go tokenizer is deleted; two implementations could drift, which
is what the MCP/HTTP parity requirement forbids.
- db.py passed 500 lines, so the row codec moved to row_codec.py.
- Tooling: installed the repo's own pyright. pyrightconfig.json had no
exclude and swept the downloaded corpus (1171 errors); with excludes the
count is 62, all pre-existing (evals 48, profiles 4, training 4,
ingestion 3, runtime 2, cartridge_assets 1). backend/pyproject.toml
carried a second [tool.pyright] that shadowed the root config; deleted.
- bench/scale_check.py measures the product search at corpus size. The
100k-atom p50 number is not written until it is measured on an idle
machine.
Verification: 112 pytest passed; go build/vet/test clean, gofmt clean;
ruff clean on every touched file; mypy clean on the 14 touched Python
files; repo pyright 62 errors, unchanged per-file ledger.
|