ZacharyZhang-NY
77219c930b
Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.
- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
gold and query, gold observed after as_of, and manifest digest drift
17 lines
323 B
TOML
17 lines
323 B
TOML
[tool.pyright]
|
|
venvPath = "."
|
|
venv = ".venv"
|
|
extraPaths = ["."]
|
|
pythonVersion = "3.12"
|
|
include = ["bench", "backend/src"]
|
|
|
|
[tool.mypy]
|
|
python_executable = ".venv/bin/python"
|
|
mypy_path = "."
|
|
python_version = "3.12"
|
|
files = ["bench", "backend/src"]
|
|
|
|
[[tool.mypy.overrides]]
|
|
module = "pyarrow.*"
|
|
ignore_missing_imports = true
|