ZacharyZhang-NY
709104e809
A review of 5289cb4 found the ASR report was not honest arithmetic:
- "audio s" was the last segment's end, not the media length. The real
container durations are 88.93 / 109.18 / 60.07 s, so the report understated
the audio by 48 s and every speed cell was wrong.
- Decoding was unpinned, so a rerun was a new draw: WER moved 0.242 to 0.262
and one video moved 0.278 to 0.583. Greedy single-temperature decoding with
no carry-over now answers the same way twice, and the harness runs the pass
twice to show it.
- The report claimed its WER was an upper bound. It is not a bound in either
direction: a caption that drops words also hides words the model dropped.
The denominators (201 / 296 / 36 reference words) are printed instead.
- "66 speech atoms" was bench-side arithmetic. The store now says 80 audio
atoms carrying 563 word timestamps, read back after a real ingest.
Same silent-skip bug in a second place: a text file over max_text_file_bytes
returned an empty batch with no reason. It raises FileTooLarge with its size,
and a walk records that refusal like any other.
Also fixed from the same review: the Go search tool never published the audio
modality, cli_senses carried a second json_dump, the no-audio-track branch had
a test that fed it markdown, _seconds was an empty wrapper, and the docstring
quoted a speed no report measured.
Gate root cause: bench/tests imports the bench package by name with no root
conftest, so bare pytest could not collect it at all. pythonpath in pyproject
fixes both launch styles; 212 tests pass either way.
Verification: 212 passed (bare pytest and python -m pytest), ruff clean,
mypy clean on the six touched files with a cold cache, repo-wide mypy 24/6
unchanged from HEAD, pyright 0 errors on the touched files, go build/vet/test
and gofmt clean.
80 lines
3.2 KiB
Python
80 lines
3.2 KiB
Python
"""The senses: what this machine can see and hear, and the commands that exercise them.
|
|
|
|
Each command here depends on a model snapshot the operator supplies. The report says which
|
|
snapshots are present so a claim about this install can be checked instead of believed.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Callable
|
|
from dataclasses import asdict
|
|
from pathlib import Path
|
|
|
|
import typer
|
|
|
|
from .asr import DEFERRED as SPEECH_DEFERRED
|
|
from .asr import SpeechTranscriber
|
|
from .cli_common import json_dump
|
|
from .config import Settings
|
|
from .db import Database
|
|
from .describe import DEFERRED, VisionDescriber
|
|
from .embeddings import EmbeddingService
|
|
from .visual_index import VisualIndex
|
|
|
|
|
|
def senses_report(settings: Settings, db: Database) -> dict[str, dict[str, object]]:
|
|
"""What this machine can see, and what it has to pretend about.
|
|
|
|
A missing snapshot is reported, not hidden: an unrun capability is a plan, and the status
|
|
output is where an operator learns which claims about this install are real today.
|
|
"""
|
|
visual = VisualIndex(settings, db)
|
|
describer = VisionDescriber(settings)
|
|
transcriber = SpeechTranscriber(settings)
|
|
index_present = visual.index_path.exists() and visual.map_path.exists()
|
|
return {
|
|
"picture_search": {
|
|
"model_dir": str(visual.model_dir() or ""),
|
|
"model_ready": visual.available(),
|
|
"index": "present" if index_present else "absent",
|
|
},
|
|
"picture_description": {
|
|
"model_dir": str(describer.model_dir() or ""),
|
|
"model_ready": describer.available(),
|
|
"note": "" if describer.available() else DEFERRED,
|
|
},
|
|
"speech": {
|
|
"model_dir": str(transcriber.model_dir() or ""),
|
|
"model_ready": transcriber.available(),
|
|
"note": "" if transcriber.available() else SPEECH_DEFERRED,
|
|
},
|
|
}
|
|
|
|
|
|
def register(app: typer.Typer, settings: Callable[[], Settings]) -> None:
|
|
"""Add the media commands. The settings thunk lets one patch in a test redirect every command."""
|
|
|
|
@app.command()
|
|
def transcribe(path: Path) -> None:
|
|
"""Words with the instant each was spoken; the snapshot stays the operator's to supply."""
|
|
segments = SpeechTranscriber(settings()).transcribe(path)
|
|
typer.echo(json_dump([asdict(segment) for segment in segments]))
|
|
|
|
@app.command("describe-image")
|
|
def describe_image(path: Path) -> None:
|
|
"""Say what a picture shows; names the setup step when no vision model is installed."""
|
|
typer.echo(json_dump({"image": str(path), "caption": VisionDescriber(settings()).describe(path)}))
|
|
|
|
@app.command("index-visual")
|
|
def index_visual() -> None:
|
|
"""Encode every image atom so a picture or a sentence can find it."""
|
|
database = Database(settings())
|
|
typer.echo(json_dump(VisualIndex(settings(), database).build()))
|
|
|
|
@app.command("build-dense-index")
|
|
def build_dense_index() -> None:
|
|
"""Encode every atom into the dense index the text retriever falls back to."""
|
|
index = EmbeddingService(settings()).build_index()
|
|
dimensions = int(index.vectors.shape[1]) if index.vectors.size else 0
|
|
typer.echo(json_dump({"atoms": len(index.atom_ids), "dimensions": dimensions}))
|