Measure the counterpart honestly: warm models, vectorised search, scope split
Three defects made the external baseline numbers and labels wrong. - bench/systems.py: _sentence_model rebuilt the model on every query, so dense arm latency was mostly model loading and the latency-vs-size curve was flat because a constant dominated it. Models are now cached per process. - bench/counterpart.py: visible() and search() walked 404k Python objects per query. Per-item holder/source/time columns are built once and the query path is numpy; 12.5 queries/s instead of 0.7. - bench/metrics.py: the aggregate key was named wrong_holder_confidence_unanswerable while it counts top hits that ARE from the asking holder. BASELINE.md read that backwards. Renamed to top_hit_from_holder_unanswerable and the sentence rewritten. - bench/run_external.py: the by-modality table read row['answerable'] but aggregate emits n_answerable, so the writer died after scoring finished. Each store is now scored twice, [holder scope] and [whole corpus], so the gain from the holder boundary is separable from the gain from ranking; progress lines every 500 queries. Verification: pytest bench/tests -q -> 61 passed; ruff check bench/ -> all checks passed; mypy (fresh cache) -> no issues in 25 source files.
This commit is contained in:
1 parent
c4e2982506
commit
dbe1d2cc82
8 files changed
+381
-34
No files matched your search
@@ -146,7 +146,7 @@ def _overall() -> dict[str, float]:
|
||||
"hit_at_10_wilson_lower": 0.45,
|
||||
"abstention_rate_answerable": 0.0,
|
||||
"abstention_rate_unanswerable": 0.0,
|
||||
"wrong_holder_confidence_unanswerable": 0.2,
|
||||
"top_hit_from_holder_unanswerable": 0.2,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -162,3 +162,28 @@ def test_percentile_needs_timings_and_a_valid_fraction() -> None:
|
||||
with pytest.raises(SystemExit, match="percentile must be within"):
|
||||
runner.percentile(1.5)
|
||||
assert runner.percentile(0.5) >= 0.0
|
||||
|
||||
|
||||
def test_the_dense_model_is_loaded_once_per_process(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The query path used to rebuild the model for every query and measured that as retrieval latency."""
|
||||
import sentence_transformers
|
||||
|
||||
from bench import systems
|
||||
|
||||
loaded: list[str] = []
|
||||
|
||||
class FakeModel:
|
||||
def __init__(self, path: str, **_kwargs: object) -> None:
|
||||
loaded.append(path)
|
||||
|
||||
monkeypatch.setattr(sentence_transformers, "SentenceTransformer", FakeModel)
|
||||
systems._MODELS.clear()
|
||||
model_dir = tmp_path / "dense"
|
||||
model_dir.mkdir()
|
||||
try:
|
||||
assert systems._sentence_model(model_dir) is systems._sentence_model(model_dir)
|
||||
assert loaded == [str(model_dir)]
|
||||
with pytest.raises(SystemExit, match="model not downloaded"):
|
||||
systems._sentence_model(tmp_path / "missing")
|
||||
finally:
|
||||
systems._MODELS.clear()
|
||||
Reference in new issue
Block a user