ZacharyZhang-NY
e74dafcd36
Three independently ugly bugs that all fed the same log flood:
* Electron's `WebFrameMain.send` writes every "Render frame was disposed"
to stderr via `console.error`. When the worker fans a second
agent-event past a reloading renderer, we produced thousands of
identical lines and stalled the main loop on stderr I/O. Added a
`mainFrame.detached` guard so the send is short-circuited before
Electron's own catch block fires, and installed a targeted filter in
`suppress-warnings.ts` (now actually imported from the entry point)
so the specific `console.error("Error sending from webFrameMain:"…)`
noise never reaches the terminal. Any other payload still surfaces.
* `dispatchAgentMessage` accepted the human-reader sentinel as a wake
target when an agent replied to `__board__` — `wakeAgentIfPossible`
then called `db.getAgent("__board__")`, threw, and the dispatcher
kept repeating on every send. Strip `HUMAN_SENDER_ID` at
`resolveMessageWakeTargets`, and keep a defense-in-depth guard in the
heartbeat so a stray caller (future automation fan-out) cannot
reintroduce the crash.
* Pi's skills loader rejects any SKILL.md whose frontmatter `name`
diverges from its parent directory. The four bundled `my-writing-style`
copies declared `name: writing-style-skill`, so every install emitted
a warning and dropped the skill. They also lived as orphan gitlinks
(mode 160000 without .gitmodules), and the global `*.md` gitignore
hid every other SKILL.md from version control. Fixed the frontmatter
to match the dir, converted the dangling submodules to plain trees,
and allowlisted `builtin-skills/**/*.md` so a fresh clone carries the
catalogue instead of silently losing it.
100 lines
3.2 KiB
Markdown
100 lines
3.2 KiB
Markdown
---
|
|
name: voiceover
|
|
description: Adding AI-generated voiceover to Remotion compositions using TTS
|
|
metadata:
|
|
tags: voiceover, audio, elevenlabs, tts, speech, calculateMetadata, dynamic duration
|
|
---
|
|
|
|
# Adding AI voiceover to a Remotion composition
|
|
|
|
Use ElevenLabs TTS to generate speech audio per scene, then use [`calculateMetadata`](./calculate-metadata) to dynamically size the composition to match the audio.
|
|
|
|
## Prerequisites
|
|
|
|
By default this guide uses **ElevenLabs** as the TTS provider (`ELEVENLABS_API_KEY` environment variable). Users may substitute any TTS service that can produce an audio file.
|
|
|
|
If the user has not specified a TTS provider, recommend ElevenLabs and ask for their API key.
|
|
|
|
Ensure the environment variable is available when running the generation script:
|
|
|
|
```bash
|
|
node --strip-types generate-voiceover.ts
|
|
```
|
|
|
|
## Generating audio with ElevenLabs
|
|
|
|
Create a script that reads the config, calls the ElevenLabs API for each scene, and writes MP3 files to the `public/` directory so Remotion can access them via `staticFile()`.
|
|
|
|
The core API call for a single scene:
|
|
|
|
```ts title="generate-voiceover.ts"
|
|
const response = await fetch(
|
|
`https://api.elevenlabs.io/v1/text-to-speech/${voiceId}`,
|
|
{
|
|
method: "POST",
|
|
headers: {
|
|
"xi-api-key": process.env.ELEVENLABS_API_KEY!,
|
|
"Content-Type": "application/json",
|
|
Accept: "audio/mpeg",
|
|
},
|
|
body: JSON.stringify({
|
|
text: "Welcome to the show.",
|
|
model_id: "eleven_multilingual_v2",
|
|
voice_settings: {
|
|
stability: 0.5,
|
|
similarity_boost: 0.75,
|
|
style: 0.3,
|
|
},
|
|
}),
|
|
},
|
|
);
|
|
|
|
const audioBuffer = Buffer.from(await response.arrayBuffer());
|
|
writeFileSync(`public/voiceover/${compositionId}/${scene.id}.mp3`, audioBuffer);
|
|
```
|
|
|
|
## Dynamic composition duration with calculateMetadata
|
|
|
|
Use [`calculateMetadata`](./calculate-metadata.md) to measure the [audio durations](./get-audio-duration.md) and set the composition length accordingly.
|
|
|
|
```tsx
|
|
import { CalculateMetadataFunction, staticFile } from "remotion";
|
|
import { getAudioDuration } from "./get-audio-duration";
|
|
|
|
const FPS = 30;
|
|
|
|
const SCENE_AUDIO_FILES = [
|
|
"voiceover/my-comp/scene-01-intro.mp3",
|
|
"voiceover/my-comp/scene-02-main.mp3",
|
|
"voiceover/my-comp/scene-03-outro.mp3",
|
|
];
|
|
|
|
export const calculateMetadata: CalculateMetadataFunction<Props> = async ({
|
|
props,
|
|
}) => {
|
|
const durations = await Promise.all(
|
|
SCENE_AUDIO_FILES.map((file) => getAudioDuration(staticFile(file))),
|
|
);
|
|
|
|
const sceneDurations = durations.map((durationInSeconds) => {
|
|
return durationInSeconds * FPS;
|
|
});
|
|
|
|
return {
|
|
durationInFrames: Math.ceil(sceneDurations.reduce((sum, d) => sum + d, 0)),
|
|
};
|
|
};
|
|
```
|
|
|
|
The computed `sceneDurations` are passed into the component via a `voiceover` prop so the component knows how long each scene should be.
|
|
|
|
If the composition uses [`<TransitionSeries>`](./transitions.md), subtract the overlap from total duration: [./transitions.md#calculating-total-composition-duration](./transitions.md#calculating-total-composition-duration)
|
|
|
|
## Rendering audio in the component
|
|
|
|
See [audio.md](./audio.md) for more information on how to render audio in the component.
|
|
|
|
## Delaying audio start
|
|
|
|
See [audio.md#delaying](./audio.md#delaying) for more information on how to delay the audio start.
|