Stop the prose from drifting away from the report it cites
A second review found three numbers left behind when the report was regenerated with pinned decoding: - the transcriber docstring claimed 1.7 s for 88.9 s of speech; the report says 1.39 s, and 1.7 came from the unpinned run. The docstring now names no number, so it cannot go stale, and it stops calling container seconds speech with voice activity detection off. - the checklist still quoted 3.55 s and 61.6x / 73.0x / 97.7x; the frozen report says 3.26 s and 63.9x / 82.4x / 112.2x. - the checklist still said 31 speech segments for one mp4; with greedy decoding that video yields 38, which is where the store's 80 atoms come from (38 + 39 + 3). The report also credited the store with timing invariants that were measured on the model's answer. The harness now recomputes them from the words as stored (out_of_order_in_store, before_their_segment_in_store) and says so. Verification: 212 passed, ruff clean, mypy clean on both touched files with a cold cache, pyright 0 errors.
This commit is contained in:
1 parent
836a4908e1
commit
10a39c5125
3 files changed
+15
-5
No files matched your search
+2
-2
@@ -160,7 +160,7 @@
|
||||
- 实测证据(`bench/reports/ASR.md`,`.venv/bin/python -m bench.asr_check` 生成):3 段 Video-MME 真视频
|
||||
**258.18 秒**音频(容器时长 88.93 / 109.18 / 60.07),对着第三方字幕 WER **0.242**——分母是
|
||||
**533 个参考词**(听到 563 个词),三行各自的参考词数是 201 / 296 / 36。CPU int8 4 线程
|
||||
3.55 秒转完,逐行 61.6× / 73.0× / 97.7× 实时。走产品路径摄入后从库里读回:**80 个 audio atom**、
|
||||
3.26 秒转完,逐行 63.9× / 82.4× / 112.2× 实时。走产品路径摄入后从库里读回:**80 个 audio atom**、
|
||||
**563 个词级时间戳**、0 个时序倒错、0 个早于自己所属片段(总共 133 个 atom)。模型
|
||||
`Systran/faster-whisper-tiny`(78.2MB,model.bin sha256 dcb76c6586fc06cb…),远低于 500MB 上限。
|
||||
- 可复现性是量出来的:解码参数钉死 `beam_size=1, temperature=(0.0,), condition_on_previous_text=False,
|
||||
@@ -183,7 +183,7 @@
|
||||
- 同一个静默毛病还有一处被 review 抓出来:`_ingest_text` 对超过 `max_text_file_bytes`(10 MiB)
|
||||
的文件也是直接返回空 batch。本仓库自己的语料里就有 277MB 的 `longmemeval_s_cleaned.json`。
|
||||
现在抛 `FileTooLarge`(带字节数和上限),扫描照样记进 `skipped`。
|
||||
- 视频现在除了帧 atom 还会产出 speech atom(同一个 mp4:18 帧 + 31 段语音)。
|
||||
- 视频现在除了帧 atom 还会产出 speech atom(同一个 mp4:18 帧 + 38 段语音,即 38 个 audio atom)。
|
||||
- 顺手把 p2-4 提前做了:`ingestion.py` 748 行超了 500 规则,拆成
|
||||
`ingestion_common.py`(102) / `ingestion_media.py`(ABC mixin, 278) / `ingestion_repository.py`(321) /
|
||||
`ingestion.py`(177);`cli.py` 544 行也超了,按已有的 `cli_transactive.register` 模式拆出
|
||||
|
||||
@@ -115,7 +115,7 @@ def _segment(segment: Any) -> SpeechSegment:
|
||||
|
||||
|
||||
class SpeechTranscriber:
|
||||
"""Whisper on CTranslate2: 88.9 seconds of speech in 1.7 seconds of CPU int8 (bench/reports/ASR.md)."""
|
||||
"""Whisper on CTranslate2, greedy on CPU int8; bench/reports/ASR.md carries the measured speed."""
|
||||
|
||||
def __init__(self, settings: Settings | None = None) -> None:
|
||||
self.settings = settings or get_settings()
|
||||
|
||||
+12
-2
@@ -148,11 +148,20 @@ def speech_atoms_in_the_store(videos: list[Path]) -> dict[str, Any]:
|
||||
service.ingest_path(video)
|
||||
atoms = database.list_atoms()
|
||||
audio = [atom for atom in atoms if atom.modality == Modality.audio]
|
||||
out_of_order = 0
|
||||
before_their_segment = 0
|
||||
for atom in audio:
|
||||
segment_start = float(atom.metadata.get("start_seconds", 0.0))
|
||||
starts = [float(word["start"]) for word in atom.metadata.get("words", [])]
|
||||
out_of_order += sum(1 for first, second in pairwise(starts) if second < first)
|
||||
before_their_segment += sum(1 for start in starts if start < segment_start - 0.01)
|
||||
return {
|
||||
"atoms_all": len(atoms),
|
||||
"speech_atoms": len(audio),
|
||||
"word_timestamps": sum(len(atom.metadata.get("words", [])) for atom in audio),
|
||||
"sources": len({atom.source_id for atom in audio}),
|
||||
"out_of_order_in_store": out_of_order,
|
||||
"before_their_segment_in_store": before_their_segment,
|
||||
}
|
||||
|
||||
|
||||
@@ -265,8 +274,9 @@ def write_markdown(report: dict[str, Any]) -> None:
|
||||
(
|
||||
f"Ingested through the product path, the store holds {totals['speech_atoms']} audio atoms from "
|
||||
f"{totals['sources']} sources carrying {totals['word_timestamps']} word timestamps "
|
||||
f"({totals['out_of_order']} out of order, {totals['before_their_segment']} before their own segment) "
|
||||
f"out of {totals['atoms_all']} atoms in total."
|
||||
f"({totals['out_of_order_in_store']} out of order, "
|
||||
f"{totals['before_their_segment_in_store']} before their own segment) out of {totals['atoms_all']} "
|
||||
"atoms in total, checked on the words as stored rather than on the model's answer."
|
||||
),
|
||||
"",
|
||||
(
|
||||
|
||||
Reference in new issue
Block a user