Add baseline harness: ingest, metrics, ablation arms, external counterpart, audit
- bench/ingest_baseline.py drives the unmodified product IngestionService over all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and memory ceilings are set explicitly: ONNX Runtime defaulted to every core (95 threads/worker, load 490) and its CPU arena kept freed pages mapped, which OOM-killed workers. - bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance baseline. No model, no product import, so the scorer cannot grade itself. - bench/systems.py + bench/run.py: the five ablation arms over the merged DB, latency percentiles, scale curve, and a top-10 equivalence check against the live RetrievalService. - bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store and Zep-style temporal store as the external reference point. - bench/audit.py: enforces single-attempt scoring, frozen n, a chance row beside every score, and producer != verifier. It already flags the product's retry-until-target loop at backend/src/ely_eye/evals.py:2351. - backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable (PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
This commit is contained in:
1 parent
77219c930b
commit
fc0698c242
27 files changed
+2589
-24
No files matched your search
+1
-1
@@ -217,7 +217,7 @@ def build() -> tuple[list, list, dict]:
|
||||
def main() -> None:
|
||||
corpus, queries, report = build()
|
||||
corpus_count, corpus_sha = write_jsonl(SETS / "corpus" / "code.jsonl", corpus)
|
||||
query_count, query_sha = write_jsonl(SETS / "code.jsonl", queries)
|
||||
_query_count, query_sha = write_jsonl(SETS / "code.jsonl", queries)
|
||||
report |= {"corpus_records": corpus_count, "corpus_sha256": corpus_sha, "queries_sha256": query_sha}
|
||||
(SETS / "reports").mkdir(parents=True, exist_ok=True)
|
||||
(SETS / "reports" / "code.json").write_text(
|
||||
|
||||
Reference in new issue
Block a user