ZacharyZhang-NY
fc0698c242
- bench/ingest_baseline.py drives the unmodified product IngestionService over all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and memory ceilings are set explicitly: ONNX Runtime defaulted to every core (95 threads/worker, load 490) and its CPU arena kept freed pages mapped, which OOM-killed workers. - bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance baseline. No model, no product import, so the scorer cannot grade itself. - bench/systems.py + bench/run.py: the five ablation arms over the merged DB, latency percentiles, scale curve, and a top-10 equivalence check against the live RetrievalService. - bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store and Zep-style temporal store as the external reference point. - bench/audit.py: enforces single-attempt scoring, frozen n, a chance row beside every score, and producer != verifier. It already flags the product's retry-until-target loop at backend/src/ely_eye/evals.py:2351. - backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable (PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
63 lines
3.3 KiB
Markdown
63 lines
3.3 KiB
Markdown
# bench — the measurement harness
|
|
|
|
Everything here measures the product as it is. No arm may read a benchmark gold label.
|
|
|
|
## run order
|
|
|
|
```bash
|
|
python -m bench.fetch_corpus # 19 GB of third-party material -> bench/corpus + ~/ely_eye_bench_data
|
|
python -m bench.build_sets # freeze the 6 query sets -> bench/sets/
|
|
python -m bench.validate_sets # schema + gold + scope + modality checks (b-2 gate)
|
|
python -m bench.fetch_models # dense + rerank models used by the ablation arms
|
|
python -m bench.ingest_baseline # the unmodified product ingesting the whole corpus (16 -> 8 workers)
|
|
python -m bench.run # five-arm ablation + latency/scale -> bench/reports/BASELINE.md
|
|
python -m pytest bench/tests -q # metric definitions, hand-computed
|
|
```
|
|
|
|
`bench/runs/` and `bench/corpus/` are gitignored; `bench/corpus/manifest.jsonl` is tracked so the
|
|
corpus is reproducible from the manifest alone.
|
|
|
|
## the arms (b-3)
|
|
|
|
| arm | what it is |
|
|
| --- | --- |
|
|
| `bm25_only` | the product's sparse signal alone |
|
|
| `dense_only` | the product's dense signal alone (product flag is off today) |
|
|
| `graph_only` | the product's graph signal alone — `atom.relations` is never populated, so it is a constant |
|
|
| `no_rerank` | the product's exact 0.55 / 0.35 / 0.10 fusion |
|
|
| `fusion_rerank` | the same fusion plus a cross-encoder rerank, the upper bound of the current design |
|
|
|
|
`python -m bench.run` also drives the **unmodified** `RetrievalService.search` over the merged
|
|
database and reports its latency plus how far its top-10 differs from the fast BM25 path.
|
|
|
|
## known environment artifact
|
|
|
|
pi-lens' pyright reports `Import "bench.sets_common" could not be resolved` (and the same for
|
|
`bench.metrics`, `bench.systems`, `ely_eye.*`, `cv2`, `onnxruntime`, `pypdf.errors`) even though
|
|
every one of those imports works. It is not a repo defect:
|
|
|
|
- `effective_config` on any bench file returns `0 config file(s)` with `resolvedCwd` at the repo
|
|
root, so neither `pyproject.toml [tool.pyright]` nor `pyrightconfig.json` (both of which carry
|
|
`extraPaths: [".", "backend/src"]`) is being read by that runner.
|
|
- Runtime proof, run from the repo root with `.venv` active:
|
|
|
|
```bash
|
|
.venv/bin/python -c "import bench.sets_common, bench.metrics, bench.systems, bench.run, bench.ingest_baseline, \
|
|
bench.build_mmlongbench, ely_eye.config, ely_eye.db, cv2, onnxruntime, pypdf.errors; print('runtime proof OK')"
|
|
```
|
|
|
|
Treat that command as the check of record for import health; treat pyright's *type* findings
|
|
(it has caught real defects here: an `AtomRunner` typo, missing `strict=True` on `zip`,
|
|
`dict[str, object]` over-narrowing) as separate and worth fixing.
|
|
|
|
## corpus facts that constrain the numbers
|
|
|
|
- 50,789 sources / 5,308 queries (2,928 answerable, 2,380 unanswerable), all third-party.
|
|
- Video-MME ships mp4s for 129 of 900 videos; the set uses those 129 (387 questions).
|
|
- Video-MME has no temporal gold, so its gold region is the whole video — recorded in the set manifest.
|
|
- ScreenSpot-v2 boxes are `[x, y, w, h]`, verified against 755 image headers (xywh in bounds 1271/1272).
|
|
- CodeSearchNet docstrings appear verbatim inside the code, so the query text is stripped before ingest
|
|
or BM25 gets a free exact match.
|
|
- The product keys a source by `{stem[:48]}_{sha256[:16]}`, so identical bytes in two sets collapse to one
|
|
product source; `ingest_report.json` reports `shared_source_ids`.
|