Add third-party benchmark suite frozen at 50,789 sources / 5,308 queries
Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.
- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
gold and query, gold observed after as_of, and manifest digest drift
This commit is contained in:
1 parent
4cd29f91e5
commit
77219c930b
39 files changed
+61358
No files matched your search
@@ -0,0 +1,16 @@
|
||||
[tool.pyright]
|
||||
venvPath = "."
|
||||
venv = ".venv"
|
||||
extraPaths = ["."]
|
||||
pythonVersion = "3.12"
|
||||
include = ["bench", "backend/src"]
|
||||
|
||||
[tool.mypy]
|
||||
python_executable = ".venv/bin/python"
|
||||
mypy_path = "."
|
||||
python_version = "3.12"
|
||||
files = ["bench", "backend/src"]
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = "pyarrow.*"
|
||||
ignore_missing_imports = true
|
||||
Reference in new issue
Block a user