Give spoken words a memory with an instant attached

An audio file used to produce no memory at all: _ingest_file returned an empty
batch for any suffix it did not know, with no log and no reason. It now raises
UnsupportedFile when you name the file and records a SkippedFile when it was a
directory walk, so a refusal is visible either way.

Speech is measured, not promised. Over 210.5 seconds of real Video-MME audio,
faster-whisper-tiny on CPU int8 transcribed at 51x realtime and reached word
error rate 0.242 against community subtitles nobody here wrote (a caption track
drops words a transcript keeps, so that is an upper bound). 66 speech atoms now
carry 550 word-level timestamps, 0 words out of order and 0 starting before
their own segment. The snapshot is 78.2 MB, under the 500 MB cap, and the report
is bench/reports/ASR.md.

Two library facts were measured rather than assumed. faster-whisper's own audio
loader is broken against PyAV 19 (av.open got an unexpected keyword
'metadata_errors') and it declares av with no version bound, so asr.decode_audio
decodes with PyAV directly and hands the model a float32 array instead of
pinning an old av with no cp314 wheel. And av.Error does not exist on PyAV 19 -
the real base is av.error.FFmpegError - so that is the only thing caught and my
own ValueErrors keep their messages.

Files over the 500-line rule were split rather than excused: ingestion.py (748)
into ingestion_common / ingestion_media / ingestion_repository / ingestion, and
cli.py (544) into cli_senses plus cli_common, following the cli_transactive
register pattern. MediaIngest declares _source_record abstract, so a half-wired
host fails at construction instead of returning None.

Verification: 207 tests pass; ruff clean on the 13 touched files; mypy clean on
them, and repo-wide mypy fell from 29 errors to 24 (5 fixed, none added);
pyright reports 0 errors on the touched files.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 21:18:28 -04:00
1 parent 03ca35763f
commit 5289cb4b01
20 files changed
+1513 -611

No files matched your search

+3 -5
View File
@@ -19,15 +19,12 @@ import json
import re
import statistics
import subprocess
import sys
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent))
from ely_eye.config import Settings
from ely_eye.db import Database
from ely_eye.ocr import read_text_from_image
@@ -44,8 +41,9 @@ from ely_eye.storage import (
visual_token_equivalent,
)
from ely_eye.visual_index import VisualIndex
from metrics import aggregate, chance_recall_at_k, score_query
from systems import machine_facts
from bench.metrics import aggregate, chance_recall_at_k, score_query
from bench.systems import machine_facts
ROOT = Path(__file__).resolve().parent
CORPUS = ROOT / "corpus" / "OS-Copilot" / "ScreenSpot-v2"