Measure the counterpart honestly: warm models, vectorised search, scope split
Three defects made the external baseline numbers and labels wrong. - bench/systems.py: _sentence_model rebuilt the model on every query, so dense arm latency was mostly model loading and the latency-vs-size curve was flat because a constant dominated it. Models are now cached per process. - bench/counterpart.py: visible() and search() walked 404k Python objects per query. Per-item holder/source/time columns are built once and the query path is numpy; 12.5 queries/s instead of 0.7. - bench/metrics.py: the aggregate key was named wrong_holder_confidence_unanswerable while it counts top hits that ARE from the asking holder. BASELINE.md read that backwards. Renamed to top_hit_from_holder_unanswerable and the sentence rewritten. - bench/run_external.py: the by-modality table read row['answerable'] but aggregate emits n_answerable, so the writer died after scoring finished. Each store is now scored twice, [holder scope] and [whole corpus], so the gain from the holder boundary is separable from the gain from ranking; progress lines every 500 queries. Verification: pytest bench/tests -q -> 61 passed; ruff check bench/ -> all checks passed; mypy (fresh cache) -> no issues in 25 source files.
This commit is contained in:
1 parent
c4e2982506
commit
dbe1d2cc82
8 files changed
+381
-34
No files matched your search
+18
-6
@@ -168,13 +168,25 @@ class DenseIndex:
|
||||
return np.asarray(self._vectors @ vector[0], dtype=np.float64)
|
||||
|
||||
|
||||
def _sentence_model(model_dir: Path) -> Any:
|
||||
"""sentence-transformers ships no stubs, so the model is typed Any rather than a fake object."""
|
||||
from sentence_transformers import SentenceTransformer
|
||||
_MODELS: dict[str, Any] = {}
|
||||
|
||||
if not model_dir.is_dir():
|
||||
raise SystemExit(f"{model_dir}: model not downloaded, run python -m bench.fetch_models first")
|
||||
return SentenceTransformer(str(model_dir), local_files_only=True)
|
||||
|
||||
def _sentence_model(model_dir: Path) -> Any:
|
||||
"""Load a sentence-transformers model once per process.
|
||||
|
||||
The query path called this per query, and constructing the model from disk cost
|
||||
more than the encode itself: dense arm p50 was 1.7s of which most was loading, and
|
||||
the latency-vs-size curve came out flat because a constant dominated it.
|
||||
sentence-transformers ships no stubs, so the model is typed Any rather than a fake object.
|
||||
"""
|
||||
key = str(model_dir)
|
||||
if key not in _MODELS:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
if not model_dir.is_dir():
|
||||
raise SystemExit(f"{model_dir}: model not downloaded, run python -m bench.fetch_models first")
|
||||
_MODELS[key] = SentenceTransformer(key, local_files_only=True)
|
||||
return _MODELS[key]
|
||||
|
||||
|
||||
def top_sources(scores: np.ndarray, atoms: Sequence[Atom], top_k: int) -> list[str]:
|
||||
|
||||
Reference in new issue
Block a user