If your product already voices users or characters with ElevenLabs, you have settled on voices, models and pronunciation fixes, and you pay for them. Some avatar APIs then want to synthesize the speech again in their own TTS, which means a second voice to tune and a second bill for the same words. The cleaner setup is to keep ElevenLabs as the voice and use the avatar API only for the video.
This guide does that with PitchTwin's audio mode: generate speech with timestamps in ElevenLabs, hand the audio and its alignment to PitchTwin, and get back an MP4 plus word timings.
Two ways to voice an avatar
PitchTwin generations come in two modes:
| Text / script mode | Audio mode | |
|---|---|---|
| Who makes the speech | PitchTwin, in the avatar's cloned voice | You (ElevenLabs, a recording, any TTS) |
| Request field | text or script | audio: { url, alignment? } |
| Price | 1.5 tokens per second of video | 1 token per second of video |
| Per minute (Starter → Scale pack) | $1.35 → $1.12 | $0.90 → $0.74 |
| Word timings | Always | From the alignment you send |
| Voice options | voice.speed, voice.language | Set them in ElevenLabs |
Text mode is simpler: one API, one bill, and the voice is cloned from the avatar's own clip when it has at least 10 seconds of speech. Audio mode makes sense when you already have an ElevenLabs setup you trust, need its full set of voice controls, or want to reuse the same audio elsewhere. In audio mode you pay ElevenLabs for the speech separately, and the PitchTwin side costs a third less per second. Full numbers are on the pricing page.
An avatar created without a voice sample has has_voice: false and only accepts audio mode, which is fine for this flow.
Step 1: speech with timestamps from ElevenLabs
ElevenLabs' Text to Speech with timestamps endpoint, POST /v1/text-to-speech/{voice_id}/with-timestamps, returns JSON instead of a raw audio stream:
audio_base64: the audio, base64-encoded.alignment:characters,character_start_times_secondsandcharacter_end_times_secondsfor the text you sent.normalized_alignment: the same for the normalized text (for example, numbers spelled out).
The output format is a query parameter (output_format, default mp3_44100_128), and the model defaults to eleven_multilingual_v2. Authentication is the xi-api-key header.
const XI = "https://api.elevenlabs.io/v1";
type Alignment = {
characters: string[];
character_start_times_seconds: number[];
character_end_times_seconds: number[];
};
export async function speak(voiceId: string, text: string) {
const res = await fetch(
`${XI}/text-to-speech/${voiceId}/with-timestamps?output_format=mp3_44100_128`,
{
method: "POST",
headers: {
"xi-api-key": process.env.ELEVENLABS_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ text, model_id: "eleven_multilingual_v2" }),
}
);
if (!res.ok) throw new Error(`ElevenLabs ${res.status}: ${await res.text()}`);
const data = await res.json();
return {
audio: Buffer.from(data.audio_base64, "base64"),
alignment: data.alignment as Alignment | null,
};
}
Use alignment, not normalized_alignment. Its characters match the text you wrote, so the word timings you get back will match your script, punctuation included.
Step 2: upload the audio to PitchTwin
The audio.url field accepts any https URL. If the file already sits on your CDN, pass that link and skip this step. Otherwise upload it: POST /uploads returns a one-time URL to PUT the file to (up to 50 MB), and you reference it afterwards as upload:<file_id>.
const PT = "https://pitchtwin.ai/api/v1";
async function pt(path: string, init: RequestInit = {}) {
const res = await fetch(PT + path, {
...init,
headers: {
Authorization: `Bearer ${process.env.PITCHTWIN_KEY}`,
"Content-Type": "application/json",
},
});
const body = await res.json();
if (!res.ok) throw new Error(`${body.error?.code}: ${body.error?.message}`);
return body;
}
export async function uploadAudio(audio: Buffer): Promise<string> {
const up = await pt("/uploads", { method: "POST" });
const put = await fetch(up.upload_url, {
method: "PUT",
headers: { "Content-Type": "audio/mpeg" },
body: audio,
});
if (!put.ok) throw new Error(`upload failed: ${put.status}`);
return `upload:${up.file_id}`;
}
Step 3: render the video
Pass the ElevenLabs alignment through unchanged. PitchTwin accepts the /with-timestamps format as is, checks that the timings fit inside the audio, and turns them into word timings.
export async function renderLine(avatarId: string, voiceId: string, text: string) {
const { audio, alignment } = await speak(voiceId, text);
const url = await uploadAudio(audio);
return pt("/generations", {
method: "POST",
body: JSON.stringify({
avatar_id: avatarId,
audio: alignment ? { url, alignment } : { url },
webhook_url: "https://your.app/hooks/pitchtwin",
}),
}); // → 202 { id, status: "queued", … }
}
The response is 202 with status: "queued". When rendering finishes, PitchTwin POSTs a signed generation.succeeded event to your webhook_url, or you poll GET /generations/:id. The result includes video_url (MP4), audio_url, duration_ms, tokens_charged and words. The webhook side is covered in Async video generation with webhooks.
Why send the alignment at all
Alignment is optional. The video renders the same without it. What you lose is the words array: in audio mode it is built from your alignment, so without one it comes back empty. Word timings are what you need for captions, word highlighting and search inside videos, and Captions from word timestamps shows how to turn them into WebVTT or SRT.
If you send audio from somewhere other than ElevenLabs, PitchTwin also accepts a simpler shape: { characters, start_times, end_times }, with times in seconds and one entry per character, spaces included.
Long scripts and multiple lines
Audio mode takes up to 10 minutes of audio per generation. For longer content, or for scripts you assemble from several parts, you have two options:
- One generation per section. Simple, and each section can be re-rendered on its own when the script changes. This suits training modules and onboarding flows.
- One audio file for the whole video. If you synthesize in chunks and concatenate the audio yourself, you also have to merge the alignments: shift each chunk's start and end times by the total duration of the chunks before it. ElevenLabs'
previous_textandnext_textrequest fields give each chunk context from its neighbours, which helps intonation stay consistent across the joins.
Either way, keep the avatar the same across sections. Each avatar is pinned to its engine version, so a video rendered next month matches one rendered today.
Languages
Because the voice comes from your audio, the avatar speaks whatever language ElevenLabs produced. That makes audio mode a natural fit for localization: one avatar, many voiced tracks. Multilingual AI avatar videos: TTS vs dubbed audio compares this to letting PitchTwin voice each language.
Practical notes
- Consent. PitchTwin's Terms require every person in an avatar to have agreed to it. If the ElevenLabs voice is a clone of a real person, get their consent for the voice as well.
- Keys. Keep both the ElevenLabs and PitchTwin keys on your server.
- Errors. An alignment whose arrays differ in length or run past the end of the audio is rejected with
invalid_alignment. A402 insufficient_tokensmeans the account needs a top-up. Audio mode is charged up front for the exact audio length, and failed generations are refunded in full automatically. - Signed links.
video_urlandaudio_urlexpire after 24 hours, so copy the MP4 to your own storage.
Try it
Sign up for 60 free tokens (no card), create an avatar from a short clip, and run renderLine with one of your ElevenLabs voices. The API docs cover the audio mode fields in full.