Add third-party benchmark suite frozen at 50,789 sources / 5,308 queries

Browse Source

Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.

- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
  annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
  corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
  query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
  gold and query, gold observed after as_of, and manifest digest drift

...

This commit is contained in:

ZacharyZhang-NY committed 2026-10-06 12:05:43 -04:00

1 parent 4cd29f91e5

commit 77219c930b

39 files changed

+61358

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bench/sets/code.jsonl

+2474

File diff suppressed because it is too large. Load diff