Many people watch video with the sound off, especially in feeds and inboxes, so videos your product generates usually need captions. Running speech-to-text on every output adds a dependency, costs money and can mishear names. If the video API already returns word timestamps, you can build accurate captions with a few dozen lines of code.
This post turns the words array from a PitchTwin generation into WebVTT and SRT files, then shows how to play them in a browser or burn them into the MP4.
What you get from the API
A succeeded generation from GET /generations/:id (or the generation.succeeded webhook) includes video_url, audio_url, duration_ms and words, the word-level timings of the final soundtrack:
{
"status": "succeeded",
"duration_ms": 5120,
"words": [
{ "text": "Hi", "start": 0.0, "end": 0.18 },
{ "text": "Mark,", "start": 0.21, "end": 0.55 },
{ "text": "saw", "start": 0.95, "end": 1.1 }
]
}
Times are in seconds from the start of the video. Punctuation stays attached to the word, which helps with line breaks.
Where the timings come from depends on the mode:
- Text or script mode. PitchTwin voices the text in the avatar's cloned voice, so timings come from that synthesis, including any
pause_after_mssilences you added. See Writing scripts for AI avatars for how pauses work. - Audio mode. You bring the audio and can send a character alignment with it (the ElevenLabs
/with-timestampsformat works as is). Thewordsarray is built from that alignment, so if you don't send one, it comes back empty. Using your own ElevenLabs audio walks through that flow.
Save words when you save the video. The video_url is a signed link that expires after 24 hours, but the timings are plain data you can keep forever. Async video generation with webhooks covers where that fits in your pipeline.
Grouping words into cues
A caption file is a list of cues, each with a start, an end and a line of text. One cue per word flickers and is hard to read, and one cue per sentence can run too long. A simple greedy grouping works well. Start a new cue when:
- Adding the next word would exceed a character limit (42 is a common choice for one line).
- The cue would run longer than a few seconds.
- There is a noticeable gap before the next word, such as a scripted pause.
- The previous word ends a sentence.
type Word = { text: string; start: number; end: number };
type Cue = { start: number; end: number; text: string };
const LIMITS = { maxChars: 42, maxDuration: 3.5, maxGap: 0.6 };
export function buildCues(words: Word[], limits = LIMITS): Cue[] {
const cues: Cue[] = [];
let line: Word[] = [];
const flush = () => {
if (!line.length) return;
cues.push({
start: line[0].start,
end: line[line.length - 1].end,
text: line.map((w) => w.text).join(" "),
});
line = [];
};
for (const w of words) {
if (line.length) {
const first = line[0];
const last = line[line.length - 1];
const chars = line.reduce((n, x) => n + x.text.length + 1, 0) + w.text.length;
if (
chars > limits.maxChars ||
w.end - first.start > limits.maxDuration ||
w.start - last.end > limits.maxGap
) {
flush();
}
}
line.push(w);
if (/[.!?…]["')\]]?$/.test(w.text)) flush();
}
flush();
return cues;
}
The limits are product decisions, not rules. Vertical video with large text needs shorter lines; a training video watched on a laptop can take longer ones.
Give short cues time to be read
A one-word cue like "Congrats!" might last 0.4 seconds, which is too fast to read. Extend each cue to a minimum duration, but never past the start of the next one:
export function padCues(cues: Cue[], minDuration = 1.0): Cue[] {
return cues.map((c, i) => {
const next = cues[i + 1];
const target = Math.max(c.end, c.start + minDuration);
return { ...c, end: next ? Math.min(target, next.start) : target };
});
}
Writing WebVTT and SRT
The two formats differ in small ways. WebVTT starts with a WEBVTT header and uses a dot before milliseconds; SRT numbers each cue and uses a comma. In WebVTT cue text, &, < and > must be escaped, or a word like <b> will be read as markup.
function stamp(sec: number, sep: "." | ","): string {
const ms = Math.round(sec * 1000);
const p = (n: number, len = 2) => String(n).padStart(len, "0");
const h = Math.floor(ms / 3_600_000);
const m = Math.floor((ms % 3_600_000) / 60_000);
const s = Math.floor((ms % 60_000) / 1000);
return `${p(h)}:${p(m)}:${p(s)}${sep}${p(ms % 1000, 3)}`;
}
const escapeVtt = (s: string) =>
s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
export function toVtt(cues: Cue[]): string {
const body = cues
.map((c) => `${stamp(c.start, ".")} --> ${stamp(c.end, ".")}\n${escapeVtt(c.text)}`)
.join("\n\n");
return `WEBVTT\n\n${body}\n`;
}
export function toSrt(cues: Cue[]): string {
return (
cues
.map((c, i) => `${i + 1}\n${stamp(c.start, ",")} --> ${stamp(c.end, ",")}\n${c.text}`)
.join("\n\n") + "\n"
);
}
Putting it together:
// pt() is a small fetch wrapper that adds the Authorization header
const g = await pt(`/generations/${generationId}`);
const cues = padCues(buildCues(g.words));
await storage.put(`captions/${generationId}.vtt`, toVtt(cues), "text/vtt");
For the example above, the first cues come out as:
WEBVTT
00:00:00.000 --> 00:00:02.000
Hi Mark, saw Acme just raised.
00:00:02.100 --> 00:00:03.100
Congrats!
Showing captions
In the browser
Serve the .vtt file with the text/vtt content type and attach it with a <track> element:
<video controls src="/videos/123.mp4">
<track kind="captions" src="/captions/123.vtt" srclang="en" label="English" default>
</video>
Captions stay editable and selectable, and viewers can turn them off. If the .vtt file is on a different origin from the page, the video element needs a crossorigin attribute and the file needs CORS headers.
Burned into the MP4
For social feeds and email previews, where you don't control the player, burn the captions into the frames with ffmpeg's subtitles filter, which needs an ffmpeg build with libass:
ffmpeg -i video.mp4 -vf "subtitles=captions.srt" -c:a copy captioned.mp4
Or add them as a soft subtitle track inside the MP4 without re-encoding the video:
ffmpeg -i video.mp4 -i captions.srt -c copy -c:s mov_text captioned.mp4
Burning in re-encodes the video, so do it once in a background worker and store the result, rather than on each request.
Other things you can build from words
The same array is useful beyond captions:
- Word-by-word highlight. Render the active word in a different color, a common style in short-form video. In the browser, compare
video.currentTimeagainst each word'sstartandend. - Chapter markers or jump links for longer training videos, keyed to the first word of each section.
- Search inside videos. Index the words with their times so a search result can open the video at the right second.
- Translated captions. Translate cue text, keep the cue times. For fully localized videos in another language, generate a new video instead; see multilingual avatar video.
Edge cases to handle
| Case | What to do |
|---|---|
words is empty | Audio mode without alignment. Skip captions or run your own transcription. |
| Very long word (URL, code) | Let it exceed maxChars on its own line rather than splitting the word. |
| Cue end before start after rounding | Guard with Math.max(start, end). |
| Right-to-left languages | Test the output in your player and with your burn-in font before shipping. |
Try it
Sign up for 60 free tokens with no card, generate a short video, and feed its words into the functions above. The API docs describe the full generation object.