ZacharyZhang-NY
5289cb4b01
An audio file used to produce no memory at all: _ingest_file returned an empty batch for any suffix it did not know, with no log and no reason. It now raises UnsupportedFile when you name the file and records a SkippedFile when it was a directory walk, so a refusal is visible either way. Speech is measured, not promised. Over 210.5 seconds of real Video-MME audio, faster-whisper-tiny on CPU int8 transcribed at 51x realtime and reached word error rate 0.242 against community subtitles nobody here wrote (a caption track drops words a transcript keeps, so that is an upper bound). 66 speech atoms now carry 550 word-level timestamps, 0 words out of order and 0 starting before their own segment. The snapshot is 78.2 MB, under the 500 MB cap, and the report is bench/reports/ASR.md. Two library facts were measured rather than assumed. faster-whisper's own audio loader is broken against PyAV 19 (av.open got an unexpected keyword 'metadata_errors') and it declares av with no version bound, so asr.decode_audio decodes with PyAV directly and hands the model a float32 array instead of pinning an old av with no cp314 wheel. And av.Error does not exist on PyAV 19 - the real base is av.error.FFmpegError - so that is the only thing caught and my own ValueErrors keep their messages. Files over the 500-line rule were split rather than excused: ingestion.py (748) into ingestion_common / ingestion_media / ingestion_repository / ingestion, and cli.py (544) into cli_senses plus cli_common, following the cli_transactive register pattern. MediaIngest declares _source_record abstract, so a half-wired host fails at construction instead of returning None. Verification: 207 tests pass; ruff clean on the 13 touched files; mypy clean on them, and repo-wide mypy fell from 29 errors to 24 (5 fixed, none added); pyright reports 0 errors on the touched files.
85 lines
3.4 KiB
Python
85 lines
3.4 KiB
Python
"""The senses: what this machine can see and hear, and the commands that exercise them.
|
|
|
|
Each command here depends on a model snapshot the operator supplies. The report says which
|
|
snapshots are present so a claim about this install can be checked instead of believed.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Callable
|
|
from dataclasses import asdict
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import typer
|
|
|
|
from .asr import DEFERRED as SPEECH_DEFERRED
|
|
from .asr import SpeechTranscriber
|
|
from .config import Settings
|
|
from .db import Database
|
|
from .describe import DEFERRED, VisionDescriber
|
|
from .embeddings import EmbeddingService
|
|
from .visual_index import VisualIndex
|
|
|
|
|
|
def senses_report(settings: Settings, db: Database) -> dict[str, dict[str, object]]:
|
|
"""What this machine can see, and what it has to pretend about.
|
|
|
|
A missing snapshot is reported, not hidden: an unrun capability is a plan, and the status
|
|
output is where an operator learns which claims about this install are real today.
|
|
"""
|
|
visual = VisualIndex(settings, db)
|
|
describer = VisionDescriber(settings)
|
|
transcriber = SpeechTranscriber(settings)
|
|
index_present = visual.index_path.exists() and visual.map_path.exists()
|
|
return {
|
|
"picture_search": {
|
|
"model_dir": str(visual.model_dir() or ""),
|
|
"model_ready": visual.available(),
|
|
"index": "present" if index_present else "absent",
|
|
},
|
|
"picture_description": {
|
|
"model_dir": str(describer.model_dir() or ""),
|
|
"model_ready": describer.available(),
|
|
"note": "" if describer.available() else DEFERRED,
|
|
},
|
|
"speech": {
|
|
"model_dir": str(transcriber.model_dir() or ""),
|
|
"model_ready": transcriber.available(),
|
|
"note": "" if transcriber.available() else SPEECH_DEFERRED,
|
|
},
|
|
}
|
|
|
|
|
|
def register(app: typer.Typer, settings: Callable[[], Settings]) -> None:
|
|
"""Add the media commands. The settings thunk lets one patch in a test redirect every command."""
|
|
|
|
def json_dump(value: Any) -> str:
|
|
import json
|
|
|
|
return json.dumps(value, ensure_ascii=False, indent=2)
|
|
|
|
@app.command()
|
|
def transcribe(path: Path) -> None:
|
|
"""Words with the instant each was spoken; the snapshot stays the operator's to supply."""
|
|
segments = SpeechTranscriber(settings()).transcribe(path)
|
|
typer.echo(json_dump([asdict(segment) for segment in segments]))
|
|
|
|
@app.command("describe-image")
|
|
def describe_image(path: Path) -> None:
|
|
"""Say what a picture shows; names the setup step when no vision model is installed."""
|
|
typer.echo(json_dump({"image": str(path), "caption": VisionDescriber(settings()).describe(path)}))
|
|
|
|
@app.command("index-visual")
|
|
def index_visual() -> None:
|
|
"""Encode every image atom so a picture or a sentence can find it."""
|
|
database = Database(settings())
|
|
typer.echo(json_dump(VisualIndex(settings(), database).build()))
|
|
|
|
@app.command("build-dense-index")
|
|
def build_dense_index() -> None:
|
|
"""Encode every atom into the dense index the text retriever falls back to."""
|
|
index = EmbeddingService(settings()).build_index()
|
|
dimensions = int(index.vectors.shape[1]) if index.vectors.size else 0
|
|
typer.echo(json_dump({"atoms": len(index.atom_ids), "dimensions": dimensions}))
|