ZacharyZhang-NY
e74dafcd36
Three independently ugly bugs that all fed the same log flood:
* Electron's `WebFrameMain.send` writes every "Render frame was disposed"
to stderr via `console.error`. When the worker fans a second
agent-event past a reloading renderer, we produced thousands of
identical lines and stalled the main loop on stderr I/O. Added a
`mainFrame.detached` guard so the send is short-circuited before
Electron's own catch block fires, and installed a targeted filter in
`suppress-warnings.ts` (now actually imported from the entry point)
so the specific `console.error("Error sending from webFrameMain:"…)`
noise never reaches the terminal. Any other payload still surfaces.
* `dispatchAgentMessage` accepted the human-reader sentinel as a wake
target when an agent replied to `__board__` — `wakeAgentIfPossible`
then called `db.getAgent("__board__")`, threw, and the dispatcher
kept repeating on every send. Strip `HUMAN_SENDER_ID` at
`resolveMessageWakeTargets`, and keep a defense-in-depth guard in the
heartbeat so a stray caller (future automation fan-out) cannot
reintroduce the crash.
* Pi's skills loader rejects any SKILL.md whose frontmatter `name`
diverges from its parent directory. The four bundled `my-writing-style`
copies declared `name: writing-style-skill`, so every install emitted
a warning and dropped the skill. They also lived as orphan gitlinks
(mode 160000 without .gitmodules), and the global `*.md` gitignore
hid every other SKILL.md from version control. Fixed the frontmatter
to match the dir, converted the dangling submodules to plain trees,
and allowlisted `builtin-skills/**/*.md` so a fresh clone carries the
catalogue instead of silently losing it.
71 lines
1.9 KiB
Markdown
71 lines
1.9 KiB
Markdown
---
|
|
name: transcribe-captions
|
|
description: Transcribing audio to generate captions in Remotion
|
|
metadata:
|
|
tags: captions, transcribe, whisper, audio, speech-to-text
|
|
---
|
|
|
|
# Transcribing audio
|
|
|
|
To transcribe audio to generate captions in Remotion, you can use the [`transcribe()`](https://www.remotion.dev/docs/install-whisper-cpp/transcribe) function from the [`@remotion/install-whisper-cpp`](https://www.remotion.dev/docs/install-whisper-cpp) package.
|
|
|
|
## Prerequisites
|
|
|
|
First, the @remotion/install-whisper-cpp package needs to be installed.
|
|
If it is not installed, use the following command:
|
|
|
|
```bash
|
|
npx remotion add @remotion/install-whisper-cpp
|
|
```
|
|
|
|
## Transcribing
|
|
|
|
Make a Node.js script to download Whisper.cpp and a model, and transcribe the audio.
|
|
|
|
```ts
|
|
import path from "path";
|
|
import {
|
|
downloadWhisperModel,
|
|
installWhisperCpp,
|
|
transcribe,
|
|
toCaptions,
|
|
} from "@remotion/install-whisper-cpp";
|
|
import fs from "fs";
|
|
|
|
const to = path.join(process.cwd(), "whisper.cpp");
|
|
|
|
await installWhisperCpp({
|
|
to,
|
|
version: "1.5.5",
|
|
});
|
|
|
|
await downloadWhisperModel({
|
|
model: "medium.en",
|
|
folder: to,
|
|
});
|
|
|
|
// Convert the audio to a 16KHz wav file first if needed:
|
|
// import {execSync} from 'child_process';
|
|
// execSync('ffmpeg -i /path/to/audio.mp4 -ar 16000 /path/to/audio.wav -y');
|
|
|
|
const whisperCppOutput = await transcribe({
|
|
model: "medium.en",
|
|
whisperPath: to,
|
|
whisperCppVersion: "1.5.5",
|
|
inputPath: "/path/to/audio123.wav",
|
|
tokenLevelTimestamps: true,
|
|
});
|
|
|
|
// Optional: Apply our recommended postprocessing
|
|
const { captions } = toCaptions({
|
|
whisperCppOutput,
|
|
});
|
|
|
|
// Write it to the public/ folder so it can be fetched from Remotion
|
|
fs.writeFileSync("captions123.json", JSON.stringify(captions, null, 2));
|
|
```
|
|
|
|
Transcribe each clip individually and create multiple JSON files.
|
|
|
|
See [Displaying captions](display-captions.md) for how to display the captions in Remotion.
|