Add baseline harness: ingest, metrics, ablation arms, external counterpart, audit
- bench/ingest_baseline.py drives the unmodified product IngestionService over all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and memory ceilings are set explicitly: ONNX Runtime defaulted to every core (95 threads/worker, load 490) and its CPU arena kept freed pages mapped, which OOM-killed workers. - bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance baseline. No model, no product import, so the scorer cannot grade itself. - bench/systems.py + bench/run.py: the five ablation arms over the merged DB, latency percentiles, scale curve, and a top-10 equivalence check against the live RetrievalService. - bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store and Zep-style temporal store as the external reference point. - bench/audit.py: enforces single-attempt scoring, frozen n, a chance row beside every score, and producer != verifier. It already flags the product's retry-until-target loop at backend/src/ely_eye/evals.py:2351. - backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable (PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
This commit is contained in:
1 parent
77219c930b
commit
fc0698c242
27 files changed
+2589
-24
No files matched your search
+13
-12
@@ -26,22 +26,23 @@
|
||||
- [x] b-2.4 陷阱类别齐:知识更新 25 / 时间推理 25 / 多会话 25(LongMemEval 原生类别)+ 无答案 2380(4 种构造:cross-holder、held-out、native unanswerable、distractor)
|
||||
- 验收:`python -m bench.validate_sets`(清单原名 dataset_check/manifest_check,实作合并为一个)报 `OK: 50789 sources, 2928 answerable queries, 2380 unanswerable` 退出 0;它同时校验 gold 存在性、gold holder==query holder、`observed_at <= as_of` 时间可达性、以及每个 set 文件的 SHA256 是否仍等于冻结的 manifest(重跑两次 SHA 相同由此保证)
|
||||
- [ ] **b-3** 检索 benchmark:五路消融 + 延迟规模
|
||||
- [ ] b-3.1 指标实现:recall@k、MRR、nDCG@10,纯 Python,无模型
|
||||
- [ ] b-3.2 消融五路:BM25-only / dense-only / graph-only / no-rerank / fusion+rerank
|
||||
- [ ] b-3.3 延迟规模曲线:1k / 10k / 100k atom,p50 与 p95
|
||||
- [x] b-3.1 指标实现:recall@k、MRR、nDCG@10,纯 Python,无模型(`bench/metrics.py` + `bench/tests/test_metrics.py` 11 条:Wilson 90/100→0.8257、10/10→0.7225、0/10→0.0 手算对齐;nDCG@10 `1.5/1.6309297535714574`;chance 公式经测试修正为 `min(1.0, k/N)`)
|
||||
- [~] b-3.2 消融五路已写(`bench/systems.py`:`bm25_only / dense_only / graph_only / no_rerank / fusion_rerank`,融合权重照抄产品 `0.55/0.35/0.10`,`graph_only` 如实返回常量序因为产品从不填 `atom.relations`),14 条测试过;数字等全量 ingest
|
||||
- [~] b-3.3 延迟规模曲线已写(`bench/run.py scale_curve`,p50/p95);档位按语料实际取 `500 / 2_000 / 10_000 / 50_000` 个 source(清单原写 1k/10k/100k atom,语料冻结为 50,789 source,100k 档不存在)
|
||||
- [ ] b-3.4 **BASELINE 报告**:在改动前的代码上跑,冻结提交
|
||||
- 验收:`python -m bench.run --config bench/configs/ablation.json` 出表;每格带 n 与 Wilson 95% 下界;`bench/reports/baseline_*.json` 入库
|
||||
- 验收:`python -m bench.run --config bench/configs/ablation.json` 出表(`--config` 已接,配置优先于默认、命令行仍可覆盖);每格带 n 与 Wilson 95% 下界;`bench/reports/baseline_*.json` 入库
|
||||
- [ ] **b-4** 外部基线
|
||||
- [ ] b-4.1 仓库内 Mem0/Zep 风格同构对照:扁平 store + user/agent/run 作用域 + 无失效
|
||||
- [ ] b-4.2 同语料同硬件跑,出对照数字
|
||||
- [x] b-4.1 仓库内 Mem0/Zep 风格同构对照:扁平 store + holder 作用域 + 无失效(`bench/counterpart.py`:`split_items` 400 字切句、`(holder, sha256(text))` 去重=Mem0、`supersede` 按 (holder, subject) 时间序关闭旧条目=Zep;`visible(holder, as_of)` 只对 temporal store 生效)+ `bench/tests/test_counterpart.py` 11 条
|
||||
- [~] b-4.2 同语料同硬件跑,出对照数字:`bench/run_external.py` 已跑通管道(shard 试跑:7,560 atom → 30,230 item,Zep 关闭 642 条),全量数字等 ingest
|
||||
- [ ] b-4.3 附录:真 Mem0 跑法与参考值,标明其 `add()` 依赖 LLM 的不可比因素
|
||||
- 验收:`bench/reports/baseline_external_*.json`;对照基线与 ely-eye 同标注集同指标
|
||||
- [ ] **b-5** 反自证审计
|
||||
- [ ] b-5.1 单次尝试:禁 best-of-N 重摇
|
||||
- [ ] b-5.2 禁达标即停:跑满 n 才算分
|
||||
- [ ] b-5.3 每个分数旁必有 chance 基线行
|
||||
- [ ] b-5.4 producer ≠ verifier:判分器与被评对象不共享题目生成
|
||||
- 验收:`python -m bench.audit` 对故意注入的 best-of-N 用例报错退出
|
||||
- [x] **b-5** 反自证审计(`bench/audit.py` + `bench/tests/test_audit.py` 9 条)
|
||||
- [x] b-5.1 单次尝试:禁 best-of-N 重摇 —— AST 规则:`while` 循环的测试或 `break` 守卫里出现分数词(pass_rate/target/recall/accuracy/score/hit_rate)即违规;函数名含 `best_of/reroll/until_pass` 即违规。已抓到产品现行违规 `backend/src/ely_eye/evals.py:2351`(Visual HashHop 达标即停)
|
||||
- [x] b-5.2 禁达标即停:跑满 n 才算分 —— 报告的 `meta.queries/answerable` 必须等于冻结集 `5308/2928`,否则必须记录 `limit_per_modality`
|
||||
- [x] b-5.3 每个分数旁必有 chance 基线行 —— 报告缺 `chance.recall_at_10` 或缺任一格的 `hit_at_10_wilson_lower` 即报错
|
||||
- [x] b-5.4 producer ≠ verifier —— `bench/metrics.py` 不得 import `ely_eye/transformers/sentence_transformers/torch/bench.systems/bench.counterpart`;每条 gold 的 `origin.dataset` 必须在 `bench/corpus/manifest.jsonl` 里且 `revision` 等于冻结 revision(实测 6 个 set 全通过)
|
||||
- 验收:`python -m bench.audit --self-test` 退出 0(注入的 best-of-N 与缺 chance 两例都被拒);`python -m bench.audit --product` 现存 3 条违规退出 1(1 条产品重摇 + 2 份报告未落盘)
|
||||
- 遗留:b-1.6 挪过来的「仓库级代码检索设置」(pyparsing/regex/tokio 三仓库)尚未做成查询集,记在 P2
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user