Add baseline harness: ingest, metrics, ablation arms, external counterpart, audit

- bench/ingest_baseline.py drives the unmodified product IngestionService over
  all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and
  memory ceilings are set explicitly: ONNX Runtime defaulted to every core
  (95 threads/worker, load 490) and its CPU arena kept freed pages mapped,
  which OOM-killed workers.
- bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance
  baseline. No model, no product import, so the scorer cannot grade itself.
- bench/systems.py + bench/run.py: the five ablation arms over the merged DB,
  latency percentiles, scale curve, and a top-10 equivalence check against the
  live RetrievalService.
- bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store
  and Zep-style temporal store as the external reference point.
- bench/audit.py: enforces single-attempt scoring, frozen n, a chance row
  beside every score, and producer != verifier. It already flags the product's
  retry-until-target loop at backend/src/ely_eye/evals.py:2351.
- backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable
  (PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-06 14:36:02 -04:00
1 parent 77219c930b
commit fc0698c242
27 files changed
+2589 -24

No files matched your search

+2 -2
View File
@@ -31,7 +31,7 @@ def http_get(url: str) -> tuple[bytes, dict[str, str]]:
if urllib.parse.urlsplit(url).scheme != "https" or urllib.parse.urlsplit(url).netloc != "huggingface.co":
raise SystemExit(f"refusing non-HF url: {url}")
try:
resp = OPENER.open(urllib.request.Request(url, headers=UA), timeout=300) # noqa: S310
resp = OPENER.open(urllib.request.Request(url, headers=UA), timeout=300)
except urllib.error.HTTPError as exc:
raise SystemExit(f"HTTP {exc.code} {url}") from exc
except urllib.error.URLError as exc:
@@ -90,7 +90,7 @@ def download_to(url: str, dest: Path) -> tuple[int, str]:
digest = hashlib.sha256()
size = 0
try:
resp = OPENER.open(urllib.request.Request(url, headers=UA), timeout=600) # noqa: S310
resp = OPENER.open(urllib.request.Request(url, headers=UA), timeout=600)
except urllib.error.HTTPError as exc:
raise SystemExit(f"HTTP {exc.code} {url}") from exc
except urllib.error.URLError as exc: