ZacharyZhang-NY
77219c930b
Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.
- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
gold and query, gold observed after as_of, and manifest digest drift
24 lines
541 B
JSON
24 lines
541 B
JSON
{
|
|
"annotations": 1272,
|
|
"corpus_records": 730,
|
|
"corpus_sha256": "5bb110728ba5726f918139743612fcb0ca1a5e91cca2d396aaf4d3b022336570",
|
|
"datasets": [
|
|
"OS-Copilot/ScreenSpot-v2"
|
|
],
|
|
"holders": [
|
|
"android",
|
|
"forum",
|
|
"gitlab",
|
|
"ios",
|
|
"macos",
|
|
"shop",
|
|
"tool",
|
|
"windows"
|
|
],
|
|
"queries": 640,
|
|
"queries_answerable": 320,
|
|
"queries_cross_holder_unanswerable": 320,
|
|
"queries_sha256": "2269dd623e67cd851fb3f6b24a4556a8462a37db98e12ad79577c5830167c7c5",
|
|
"revision": "5efbb1f1b5463a575f2eb7bc30fe29e49c15f93c",
|
|
"screenshots": 730
|
|
} |