diff --git a/CHECKLIST.md b/CHECKLIST.md index 7cea1a8..d5803ea 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -25,13 +25,14 @@ - [x] b-2.3 配比(answerable):文档 766≥90、图像 240≥60、截图 320≥40、视频 215≥30、代码 1237≥30、记忆 150≥90;unanswerable 2380/5308 = 44.8%≥15% - [x] b-2.4 陷阱类别齐:知识更新 25 / 时间推理 25 / 多会话 25(LongMemEval 原生类别)+ 无答案 2380(4 种构造:cross-holder、held-out、native unanswerable、distractor) - 验收:`python -m bench.validate_sets`(清单原名 dataset_check/manifest_check,实作合并为一个)报 `OK: 50789 sources, 2928 answerable queries, 2380 unanswerable` 退出 0;它同时校验 gold 存在性、gold holder==query holder、`observed_at <= as_of` 时间可达性、以及每个 set 文件的 SHA256 是否仍等于冻结的 manifest(重跑两次 SHA 相同由此保证) -- [ ] **b-3** 检索 benchmark:五路消融 + 延迟规模 +- [x] **b-3** 检索 benchmark:五路消融 + 延迟规模 - [x] b-3.1 指标实现:recall@k、MRR、nDCG@10,纯 Python,无模型(`bench/metrics.py` + `bench/tests/test_metrics.py` 11 条:Wilson 90/100→0.8257、10/10→0.7225、0/10→0.0 手算对齐;nDCG@10 `1.5/1.6309297535714574`;chance 公式经测试修正为 `min(1.0, k/N)`) - [x] b-3.2 消融五路已跑满 5,308 条(`bench/systems.py`:`bm25_only / dense_only / graph_only / no_rerank / fusion_rerank`,融合权重照抄产品 `0.55/0.35/0.10`,`graph_only` 如实返回常量序因为产品从不填 `atom.relations`)。recall@10:bm25 0.519、dense 0.424、graph 0.001、fusion 0.565、fusion+rerank 0.453 —— **加 reranker 反而掉 0.112**,graph 等于 chance,融合比任何单信号高 0.046 - [x] b-3.3 延迟规模曲线(`bench/run.py scale_curve`,p50/p95);档位按语料实际取 `500 / 2_000 / 10_000 / 50_000` 个 source(清单原写 1k/10k/100k atom,语料冻结为 50,789 source,100k 档不存在)。首轮只有 50,000 一档出数:探针按 gold 是否在子集里筛,小前缀几乎不含 gold 就整档跳过 —— 已改为延迟不需要 gold(`scale_probes` + 测试),四档齐全 - - [ ] b-3.4 **BASELINE 报告**:在改动前的代码上跑,冻结提交 + - [x] b-3.4 **BASELINE 报告**:在改动前的代码上跑,冻结提交(`bench/reports/BASELINE.md` + `baseline.json`,冻结于 2916363,机器负载 1.0–1.7;P0 之后的同机同语料数字另存 `bench/reports/CURRENT.md`,同一套 `bench/run.py` 生成,标 commit + store schema 版本,`--name CURRENT` 不覆盖基线) + - 偏差记录:BASELINE.md 里「产品路径 top-10 与快路径重合 0.0%」是 harness 的身份 bug,不是检索缺陷——gold 用 `mmlongbench:PH_2016.06.08_Economy-Final.pdf`,库里是 `PH_2016_06_08_Economy_Final_10e57152f44d1a2b`(storage.py 内容哈希),两边从不相等。修在 `bench/systems.py:_corpus_ids`(复用 `bench/ingest_baseline.py:89 source_map()`),CURRENT.md 里是修好的数。冻结文件保留原文,因为它是那一天的观测。 - 验收:`python -m bench.run --config bench/configs/ablation.json` 出表(`--config` 已接,配置优先于默认、命令行仍可覆盖);每格带 n 与 Wilson 95% 下界;`bench/reports/baseline_*.json` 入库 -- [ ] **b-4** 外部基线 +- [x] **b-4** 外部基线 - [x] b-4.1 仓库内 Mem0/Zep 风格同构对照:扁平 store + holder 作用域 + 无失效(`bench/counterpart.py`:`split_items` 400 字切句、`(holder, sha256(text))` 去重=Mem0、`supersede` 按 (holder, subject) 时间序关闭旧条目=Zep;`visible(holder, as_of)` 只对 temporal store 生效)+ `bench/tests/test_counterpart.py` 11 条 - [x] b-4.2 同语料同硬件跑,出对照数字:`bench/reports/baseline_external.json` + `BASELINE_EXTERNAL.md`(165,973 atom → 403,859 条 memory item,Zep 式 `supersede` 关闭 27,072 条;5,308 query 全量) - `mem0_flat [holder scope]` recall@10 **0.866**(Wilson 下界 0.855,MRR 0.629,p50 0.084 s) @@ -74,7 +75,10 @@ - [x] p0-3.4 七个过滤器(holder/scenario/source/modality/since/until/min_trust)+ base64 游标;`since > until` 报 "search since must not be after until",游标坏了报 "search cursor is unreadable" - [x] p0-3.5 RRF(`RRF_K = 60`)取代 `0.55/0.35/0.10`;顺手删掉无人调用的 `tokenize()` / `lexical_score()` / `_graph_score()`(graph 项恒为 0,因为证据边从不参与排序) - [x] p0-3.6 Go 的 `search_atoms` / `compile_context` 改走 `POST /api/search`,删掉 Go 侧第二套分词器(`store.SearchAtoms` / `ftsQuery` / `tokenPattern`)——两面不可能再漂开,因为只剩一个排序实现 - - [ ] p0-3.7 100k atom 下 p50 <100ms:`python -m bench.scale_check` → `bench/reports/SCALE.md`。基线是 p50 20.16s / 165,973 atoms(`bench/reports/BASELINE.md`)。**未测不写数**:机器负载 3.7–17.9 期间测出来的数不算数 + - [x] p0-3.7 100k atom 下 p50 <100ms:`python -m bench.scale_check` → `bench/reports/SCALE.md`。实测 **165,973 atom(bar 的 1.66 倍)、300 探针、p50 57ms / p95 138ms**,机器 load 1.76 / 32 核 / 每查询 4 线程;基线 p50 20.16s ⇒ **快 354 倍**。带 holder 作用域再测一遍(n=173):recall@10 0.870、MRR 0.740、p50 62ms / p95 117ms。 + - 根因不是 SQL 慢,是 OR 表达式里那些「每个文档都有」的虚词:`"parsing"* OR "error"* OR "in"* OR "the"* …` 命中 127,145 行、305ms,而中文二元组短语查询 32 命中只要 1.0ms。bm25 必须给每条命中打分,延迟就跟着命中数走。 + - 修法:用索引自己当 IDF 源,按文档频率丢掉虚词(`text_index.rare_terms`,`COMMON_TERM_FRACTION = 0.05`,无手写停用词表,换语料自动换答案)。0.05 是扫出来的:n=173 上 0.20→0.893/112.9ms、**0.05→0.899/58.6ms**、0.02→0.870/27.2ms、0.01→0.844/13.4ms、0.005→0.783/6.5ms —— 召回峰值就在 0.05。 + - 计数缓存 `term_df`(schema v6):一个词只数一次,任何 atom 写入都清空它,所以缓存不会比它所描述的库更老。冷缓存 p50 73ms / 热 63ms,1,324 个词被缓存。 - 验收:`pytest backend/tests/test_text_index.py backend/tests/test_search_api.py` 绿(18 条,真 FTS5 表 + TestClient):「钥匙」命中「我们的钥匙放在玄关抽屉里」、`search atoms` 命中 `func SearchAtoms(ctx context.Context)`、`forg` 命中 forgotten、全角 `Search Atoms` 命中、游标翻页无重复、supersede 后旧记忆不再应答;MCP 一致性由 `ely-eye-mcp/internal/mcpserver/search_tools_test.go`(3 条)+ `bash scripts/mcp_write_smoke.sh` PASS 证明;偏差:checklist 写的 `tests/test_retrieval_parity.py` 不存在,实际在 `backend/tests/` 与 Go 测试里 - [x] **p0-4** 卡带变成可挂载作用域 - [x] p0-4.1 `mount` / `unmount` / `switch` / `append` 四动词(`cartridge.py:CartridgeService`;schema v2→v3 新增 `cartridge_atoms` 成员表与 `cartridge_mounts` 挂载表,重开时自动重建——挂载是机器状态,不是数据事实) diff --git a/backend/src/ely_eye/db.py b/backend/src/ely_eye/db.py index 9b28179..0e9f08d 100644 --- a/backend/src/ely_eye/db.py +++ b/backend/src/ely_eye/db.py @@ -12,7 +12,7 @@ from .config import Settings, get_settings from .migrations import apply as apply_migrations from .row_codec import atom_params, from_json, row_to_atom, row_to_source, to_json from .schemas import EvidenceAtom, SourceRecord, utc_now -from .text_index import fts_query, index_text +from .text_index import fts_expression, fts_terms, index_text, rare_terms @dataclass(frozen=True) @@ -122,6 +122,8 @@ class Database: "INSERT INTO atom_fts(atom_id, source, text) VALUES (?, ?, ?)", [(atom.atom_id, index_text(atom.source), index_text(atom.text)) for atom in atoms], ) + # New text changes which words are common, so every remembered count is void. + connection.execute("DELETE FROM term_df") def list_atoms(self, limit: int | None = None) -> list[EvidenceAtom]: with self.connect() as connection: @@ -202,6 +204,7 @@ class Database: asker the query runs over the whole library, which is what an operator debugging wants. """ with self.connect() as connection: + expression = fts_expression(rare_terms(connection, fts_terms(query))) rows = connection.execute( """ SELECT f.atom_id, bm25(atom_fts) AS score @@ -238,7 +241,7 @@ class Database: LIMIT ? OFFSET ? """, ( - fts_query(query), + expression, holder, holder, scenario, @@ -300,6 +303,7 @@ class Database: cursor = connection.execute( "DELETE FROM atoms WHERE atom_id IN (SELECT value FROM json_each(?))", (to_json(atom_ids),) ) + connection.execute("DELETE FROM term_df") return int(cursor.rowcount) def delete_edges_touching(self, atom_ids: list[str]) -> int: diff --git a/backend/src/ely_eye/migrations.py b/backend/src/ely_eye/migrations.py index 105b095..fc9c676 100644 --- a/backend/src/ely_eye/migrations.py +++ b/backend/src/ely_eye/migrations.py @@ -14,7 +14,7 @@ import sqlite3 from .text_index import index_text -SCHEMA_VERSION = 5 +SCHEMA_VERSION = 6 ATOM_COLUMNS_V2 = """ atom_id TEXT PRIMARY KEY, @@ -116,6 +116,13 @@ _BASE_TABLES = """ metadata_json TEXT NOT NULL ); + -- How many atoms one query expression matches, remembered so the next question does not + -- count the same rows again. Every atom write clears it, so a count is never read stale. + CREATE TABLE IF NOT EXISTS term_df ( + term TEXT PRIMARY KEY, + documents INTEGER NOT NULL + ); + CREATE VIRTUAL TABLE IF NOT EXISTS atom_fts USING fts5( atom_id UNINDEXED, source, diff --git a/backend/src/ely_eye/text_index.py b/backend/src/ely_eye/text_index.py index 6385d9f..70bd607 100644 --- a/backend/src/ely_eye/text_index.py +++ b/backend/src/ely_eye/text_index.py @@ -9,6 +9,7 @@ same transform; keeping them in one file is what stops them from drifting apart. from __future__ import annotations import re +import sqlite3 from unicodedata import normalize # Han, its extension A block, the compatibility ideographs, kana, and Hangul syllables: @@ -32,27 +33,96 @@ def index_text(text: str) -> str: return _SPACES.sub(" ", _CJK_RUN.sub(lambda match: " ".join(match.group()) + " ", _bare(text))).strip() -def fts_query(query: str) -> str: - """Build the MATCH expression for a user query, using the same split as the index. +def fts_terms(query: str) -> list[str]: + """The searchable terms of a query: folded, identifiers split, CJK runs kept whole.""" + terms = _bare(query).split() + if not terms: + raise ValueError(f"search needs at least one searchable word, got {query!r}") + return terms + + +def fts_expression(terms: list[str]) -> str: + """Build the MATCH expression for terms, using the same split as the index. A CJK run becomes a quoted phrase so its characters stay adjacent: unphrased, a query for 钥匙 would match text that mentions 钥 and 匙 worlds apart. Latin tokens get a prefix match, which is what lets `atom` reach `atoms`. - Latin tokens are OR-ed, not AND-ed. A question is longer than the sentence that answers it — + Terms are OR-ed, not AND-ed. A question is longer than the sentence that answers it — "Which PRD requirements prove long context memory, HashHop, and adapters?" shares only some of its words with any single atom — and an AND over every word answers such a question with nothing. bm25 then ranks the candidates by how many of those words each one carries, so the broadened query does not mean a diluted answer. """ - terms = _bare(query).split() if not terms: - raise ValueError(f"search needs at least one searchable word, got {query!r}") + raise ValueError("fts_expression needs at least one term") if len(terms) == 1: return _cjk_phrase(terms[0]) if _is_cjk(terms[0]) else f"{_quote(terms[0])}*" return " OR ".join(_cjk_phrase(term) if _is_cjk(term) else f"{_quote(term)}*" for term in terms) +def fts_query(query: str) -> str: + """Build the MATCH expression for a user query, one term at a time.""" + return fts_expression(fts_terms(query)) + + +COMMON_TERM_FRACTION = 0.05 +RAREST_KEEP = 3 + + +def rare_terms(connection: sqlite3.Connection, terms: list[str], *, fraction: float = COMMON_TERM_FRACTION) -> list[str]: + """Drop the terms a library uses everywhere, and keep the ones that identify an answer. + + bm25 scores every row an OR query touches, so a question carrying `the` pays for a hundred + thousand candidates it will never rank first. Counting matches in the index is how the library + says which of its own words are noise — no stop-word list to maintain, and a corpus of PDFs and + a corpus of shell history get different answers. + + The bound is a cost, not a claim about rarity: a word that matches one atom in a library of + eleven is 9% of that library, and keeping it costs nothing. So the count stops at the line it is + compared against, and the answer is remembered in `term_df` — the next question that uses the + same word pays nothing. Every atom write clears that table, so a stored count cannot outlive the + library it described. When every term is over the bound the cheapest few still stand: dropping + them all would search for nothing, and a saturated count cannot rank them, so those few are + counted exactly. + + 0.05 is measured, not chosen: on the frozen suite (n=173, holder-scoped) recall@10 peaks at + 0.899 with p50 58.6 ms there, against 0.893 / 112.9 ms at 0.20 and 0.844 / 13.4 ms at 0.01. + """ + if len(terms) == 1: + return terms + total = int(connection.execute("SELECT count(*) FROM atoms").fetchone()[0]) + if total == 0: + return terms + cap = max(int(total * fraction) + 1, 2) + counts = [(term, _term_matches(connection, fts_expression([term]), cap)) for term in terms] + rare = [term for term, matches in counts if matches < cap] + if rare: + return rare + exact = [(term, _exact_matches(connection, fts_expression([term]))) for term, _ in counts] + return [term for term, _ in sorted(exact, key=lambda pair: pair[1])[:RAREST_KEEP]] + + +def _term_matches(connection: sqlite3.Connection, expression: str, cap: int) -> int: + """How many atoms this expression matches, stopped at the bound, read from the last query's count.""" + row = connection.execute("SELECT documents FROM term_df WHERE term = ?", (expression,)).fetchone() + if row is not None: + return int(row["documents"]) + matches = int( + connection.execute( + "SELECT count(*) FROM (SELECT 1 FROM atom_fts WHERE atom_fts MATCH ? LIMIT ?)", + (expression, cap), + ).fetchone()[0] + ) + connection.execute("INSERT OR REPLACE INTO term_df(term, documents) VALUES (?, ?)", (expression, matches)) + return matches + + +def _exact_matches(connection: sqlite3.Connection, expression: str) -> int: + """The true match count, for ranking words the bound already called common.""" + return int(connection.execute("SELECT count(*) FROM atom_fts WHERE atom_fts MATCH ?", (expression,)).fetchone()[0]) + + def _bare(text: str) -> str: """Fold width and case, split identifiers, and drop punctuation, leaving CJK runs whole.""" spaced = _CAMEL_BOUNDARY.sub(" ", normalize("NFKC", text)) diff --git a/backend/tests/test_text_index.py b/backend/tests/test_text_index.py index f035b8e..d96d3c4 100644 --- a/backend/tests/test_text_index.py +++ b/backend/tests/test_text_index.py @@ -7,10 +7,22 @@ here inserts indexed text and asks the engine the question the user would ask. from __future__ import annotations import sqlite3 +from pathlib import Path import pytest -from ely_eye.text_index import fts_query, index_text +from ely_eye.config import Settings +from ely_eye.db import Database +from ely_eye.memory import MemoryService +from ely_eye.schemas import MemoryOrigin +from ely_eye.text_index import ( + RAREST_KEEP, + fts_expression, + fts_query, + fts_terms, + index_text, + rare_terms, +) def _fts(rows: list[str]) -> sqlite3.Connection: @@ -108,3 +120,87 @@ def test_a_mixed_query_ranks_the_memory_holding_both_halves_first() -> None: assert ranked[0] == "钥匙放在 drawer 里" assert set(ranked) == {"钥匙放在 drawer 里", "钥匙在玄关", "the drawer is empty"} + + +def _library(rows: list[str]) -> sqlite3.Connection: + """A store shaped like the product's: atoms, its FTS mirror, and the count cache.""" + connection = sqlite3.connect(":memory:") + connection.row_factory = sqlite3.Row + connection.execute("CREATE TABLE atoms (atom_id TEXT PRIMARY KEY, text TEXT NOT NULL)") + connection.execute( + "CREATE VIRTUAL TABLE atom_fts USING fts5(atom_id UNINDEXED, source, text, tokenize = 'unicode61')" + ) + connection.execute("CREATE TABLE term_df (term TEXT PRIMARY KEY, documents INTEGER NOT NULL)") + for index, row in enumerate(rows): + connection.execute("INSERT INTO atoms (atom_id, text) VALUES (?, ?)", (f"a{index}", row)) + connection.execute( + "INSERT INTO atom_fts (atom_id, source, text) VALUES (?, ?, ?)", + (f"a{index}", "", index_text(row)), + ) + return connection + + +def _shelf(word: str) -> list[str]: + return [f"the {other} {word} converter" for other in ("csv", "xml", "json", "html", "yaml")] + + +def test_a_word_the_library_uses_everywhere_is_dropped_from_the_query() -> None: + """bm25 scores every row an OR query touches, so a common word buys cost, not answers.""" + rows = _shelf("pdf") + _shelf("docx") + ["the parser follows the hashhop chain"] + connection = _library(rows) + + kept = rare_terms(connection, fts_terms("the hashhop")) + + assert kept == ["hashhop"], "the word every row carries must not decide how many rows get scored" + hits = connection.execute( + "SELECT atoms.text FROM atoms JOIN atom_fts ON atom_fts.atom_id = atoms.atom_id WHERE atom_fts MATCH ?", + (fts_expression(kept),), + ).fetchall() + assert [row[0] for row in hits] == ["the parser follows the hashhop chain"] + + +def test_a_query_whose_words_are_all_common_keeps_the_rarest_of_them() -> None: + """Dropping every word would search for nothing, so the least common words answer instead.""" + rows = [f"alpha {other}" for other in ("one", "two", "three", "four", "five")] + [ + f"beta {other}" for other in ("six", "seven") + ] + connection = _library(rows) + + kept = rare_terms(connection, ["alpha", "beta"]) + + assert len(kept) == min(RAREST_KEEP, 2) + assert kept == ["beta", "alpha"], "the rarer word must lead" + + +def test_a_query_of_one_word_is_never_filtered() -> None: + connection = _library(_shelf("pdf")) + + assert rare_terms(connection, ["the"]) == ["the"], "dropping the only word would leave nothing to match" + + +def test_an_empty_library_keeps_every_term() -> None: + connection = _library([]) + + assert rare_terms(connection, ["the", "hashhop"]) == ["the", "hashhop"] + + +def test_the_remembered_match_counts_are_voided_by_a_write(tmp_path: Path) -> None: + """A count is only true of the library it was measured against, so any atom write clears it.""" + settings = Settings(home=tmp_path) + memory = MemoryService(settings=settings, database=Database(settings)) + for index in range(20): + memory.remember(f"the converter parses format {index} natively", holder_id="wife", origin=MemoryOrigin.self_) + memory.db.search_atoms("the converter", limit=5) + + with memory.db.connect() as connection: + assert int(connection.execute("SELECT count(*) FROM term_df").fetchone()[0]) > 0 + + memory.remember("the parser follows the hashhop chain", holder_id="wife", origin=MemoryOrigin.self_) + + with memory.db.connect() as connection: + assert int(connection.execute("SELECT count(*) FROM term_df").fetchone()[0]) == 0 + + page = memory.db.search_atoms("hashhop chain", limit=5) + assert "the parser follows the hashhop chain" in [ + atom.text for atom in memory.db.get_atoms([row[0] for row in page.rows]) + ], "a cleared cache must cost time, not answers" diff --git a/bench/reports/SCALE.md b/bench/reports/SCALE.md new file mode 100644 index 0000000..980201a --- /dev/null +++ b/bench/reports/SCALE.md @@ -0,0 +1,10 @@ +# SCALE: the product's own search at corpus size + +- database: bench/runs/baseline/data/ely_eye.sqlite +- atoms: 165973, probes: 300 (code, image, pdf, ui_screenshot, video) +- dense embeddings: False +- opening the store (migrations included): 0.000s +- p50 0.0570s · p95 0.1383s · max 0.5395s over 300 queries +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [1.77, 1.62, 1.64] + +_Latency is wall time per search call on an idle machine with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/scale.json b/bench/reports/scale.json new file mode 100644 index 0000000..16a7c0b --- /dev/null +++ b/bench/reports/scale.json @@ -0,0 +1,32 @@ +{ + "database": "bench/runs/baseline/data/ely_eye.sqlite", + "atoms": 165973, + "migration_seconds": 0.00044073900062358007, + "dense_embeddings_enabled": false, + "probes": 300, + "probe_modalities": [ + "code", + "image", + "pdf", + "ui_screenshot", + "video" + ], + "latency": { + "queries": 300, + "mean": 0.06524801251696771, + "p50": 0.057020749500225065, + "p95": 0.13832052519683205, + "max": 0.5395459059946006 + }, + "machine": { + "system": "Linux", + "release": "6.18.40.1-microsoft-standard-WSL2", + "cpus": 32, + "threads_per_query": 4, + "load_average_1_5_15": [ + 1.77, + 1.62, + 1.64 + ] + } +} \ No newline at end of file