ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Freeze the ASR report against the commit that carries the code

Browse Source 
Generated by .venv/bin/python -m bench.asr_check after 709104e, so the stamp
names the code under measurement instead of its parent. Two passes agreed:
WER 0.242 both times, transcripts byte-identical.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 22:24:56 -04:00
1 parent 709104e809
commit 836a4908e1
2 files changed
+91 -50
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bench/reports/ASR.md
+16 -11
View File
@@ -1,16 +1,21 @@
# ASR: what the speech reader hears (commit 03ca35763)
# ASR: what the speech reader hears (commit 709104e80)
- generated 2026-10-08T01:05:45+00:00
- model: Systran/faster-whisper-tiny (cpu int8), model.bin sha256 dcb76c6586fc06cb…
- reference: Video-MME community subtitles (third-party captions, not verbatim transcripts)
- machine: {'system': 'Linux', 'release': '6.18.40.1-microsoft-standard-WSL2', 'cpus': 32, 'threads_per_query': 4, 'load_average_1_5_15': [1.84, 1.85, 6.99]}
- generated 2026-10-08T02:24:47+00:00
- model: Systran/faster-whisper-tiny (cpu int8, 4 threads), model.bin sha256 dcb76c6586fc06cb…
- decode: word_timestamps=True, beam_size=1, temperature=(0.0,), condition_on_previous_text=False, vad_filter=False
- reference: Video-MME community subtitles: third-party captions, not verbatim transcripts
- machine: {'system': 'Linux', 'release': '6.18.40.1-microsoft-standard-WSL2', 'cpus': 32, 'threads_per_query': 4, 'load_average_1_5_15': [9.65, 2.54, 0.88]}
| video | audio s | transcribe s | ×realtime | speech atoms | words | WER | mean word confidence |
| video | audio s | transcribe s | ×realtime | reference words | words | WER | mean word confidence |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0IdYJGBmguM | 88.9 | 1.73 | 51.3 | 31 | 218 | 0.179 | 0.879 |
| 1q-5IIyZL20 | 99.9 | 1.94 | 51.6 | 32 | 300 | 0.28 | 0.83 |
| 1tidNDIpKSY | 21.7 | 1.34 | 16.1 | 3 | 32 | 0.278 | 0.852 |
| 0IdYJGBmguM | 88.93 | 1.39 | 63.9 | 201 | 219 | 0.229 | 0.861 |
| 1q-5IIyZL20 | 109.18 | 1.33 | 82.4 | 296 | 313 | 0.253 | 0.821 |
| 1tidNDIpKSY | 60.07 | 0.54 | 112.2 | 36 | 31 | 0.222 | 0.852 |
Over 3 videos and 210.5 seconds of audio: word error rate **0.242** against captions nobody here wrote, 66 speech atoms carrying 550 word-level timestamps, 0 words out of order and 0 starting before their own segment.
Over 3 videos and 258.18 seconds of audio: word error rate **0.242** over 533 reference words (563 words heard), transcribed in 3.26 seconds.
A caption track drops words a transcript keeps, so this WER is an upper bound on the model's real error. The point of the run is not the number: before this change an audio file produced no memory at all, and every word here now has an instant attached to it.
Repeating the pass on the same machine gave WER 0.242 then 0.242; the transcripts were byte-identical. Decoding is greedy with no retry ladder, so a rerun is the same answer, not a new draw.
Ingested through the product path, the store holds 80 audio atoms from 3 sources carrying 563 word timestamps (0 out of order, 0 before their own segment) out of 133 atoms in total.
The captions are not a transcript, so this WER is not a bound in either direction: a caption that drops words also hides words the model dropped. The third row has only 36 reference words, where one word moves its rate by several points — read the denominators, not the decimals.