Engineering

ElevenLabs audio with an avatar API: a Node.js guide

Use your own ElevenLabs audio with an avatar API: generate speech with timestamps, upload it, render the video and keep word timings for captions. Node.js code.

PitchTwin team, , 6 min read

If your product already voices users or characters with ElevenLabs, you have settled on voices, models and pronunciation fixes, and you pay for them. Some avatar APIs then want to synthesize the speech again in their own TTS, which means a second voice to tune and a second bill for the same words. The cleaner setup is to keep ElevenLabs as the voice and use the avatar API only for the video.

This guide does that with PitchTwin's audio mode: generate speech with timestamps in ElevenLabs, hand the audio and its alignment to PitchTwin, and get back an MP4 plus word timings.

Two ways to voice an avatar

PitchTwin generations come in two modes:

Text / script modeAudio mode
Who makes the speechPitchTwin, in the avatar's cloned voiceYou (ElevenLabs, a recording, any TTS)
Request fieldtext or scriptaudio: { url, alignment? }
Price1.5 tokens per second of video1 token per second of video
Per minute (Starter → Scale pack)$1.35 → $1.12$0.90 → $0.74
Word timingsAlwaysFrom the alignment you send
Voice optionsvoice.speed, voice.languageSet them in ElevenLabs

Text mode is simpler: one API, one bill, and the voice is cloned from the avatar's own clip when it has at least 10 seconds of speech. Audio mode makes sense when you already have an ElevenLabs setup you trust, need its full set of voice controls, or want to reuse the same audio elsewhere. In audio mode you pay ElevenLabs for the speech separately, and the PitchTwin side costs a third less per second. Full numbers are on the pricing page.

An avatar created without a voice sample has has_voice: false and only accepts audio mode, which is fine for this flow.

Step 1: speech with timestamps from ElevenLabs

ElevenLabs' Text to Speech with timestamps endpoint, POST /v1/text-to-speech/{voice_id}/with-timestamps, returns JSON instead of a raw audio stream:

  • audio_base64: the audio, base64-encoded.
  • alignment: characters, character_start_times_seconds and character_end_times_seconds for the text you sent.
  • normalized_alignment: the same for the normalized text (for example, numbers spelled out).

The output format is a query parameter (output_format, default mp3_44100_128), and the model defaults to eleven_multilingual_v2. Authentication is the xi-api-key header.

const XI = "https://api.elevenlabs.io/v1";

type Alignment = {
  characters: string[];
  character_start_times_seconds: number[];
  character_end_times_seconds: number[];
};

export async function speak(voiceId: string, text: string) {
  const res = await fetch(
    `${XI}/text-to-speech/${voiceId}/with-timestamps?output_format=mp3_44100_128`,
    {
      method: "POST",
      headers: {
        "xi-api-key": process.env.ELEVENLABS_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ text, model_id: "eleven_multilingual_v2" }),
    }
  );
  if (!res.ok) throw new Error(`ElevenLabs ${res.status}: ${await res.text()}`);

  const data = await res.json();
  return {
    audio: Buffer.from(data.audio_base64, "base64"),
    alignment: data.alignment as Alignment | null,
  };
}

Use alignment, not normalized_alignment. Its characters match the text you wrote, so the word timings you get back will match your script, punctuation included.

Step 2: upload the audio to PitchTwin

The audio.url field accepts any https URL. If the file already sits on your CDN, pass that link and skip this step. Otherwise upload it: POST /uploads returns a one-time URL to PUT the file to (up to 50 MB), and you reference it afterwards as upload:<file_id>.

const PT = "https://pitchtwin.ai/api/v1";

async function pt(path: string, init: RequestInit = {}) {
  const res = await fetch(PT + path, {
    ...init,
    headers: {
      Authorization: `Bearer ${process.env.PITCHTWIN_KEY}`,
      "Content-Type": "application/json",
    },
  });
  const body = await res.json();
  if (!res.ok) throw new Error(`${body.error?.code}: ${body.error?.message}`);
  return body;
}

export async function uploadAudio(audio: Buffer): Promise<string> {
  const up = await pt("/uploads", { method: "POST" });
  const put = await fetch(up.upload_url, {
    method: "PUT",
    headers: { "Content-Type": "audio/mpeg" },
    body: audio,
  });
  if (!put.ok) throw new Error(`upload failed: ${put.status}`);
  return `upload:${up.file_id}`;
}

Step 3: render the video

Pass the ElevenLabs alignment through unchanged. PitchTwin accepts the /with-timestamps format as is, checks that the timings fit inside the audio, and turns them into word timings.

export async function renderLine(avatarId: string, voiceId: string, text: string) {
  const { audio, alignment } = await speak(voiceId, text);
  const url = await uploadAudio(audio);

  return pt("/generations", {
    method: "POST",
    body: JSON.stringify({
      avatar_id: avatarId,
      audio: alignment ? { url, alignment } : { url },
      webhook_url: "https://your.app/hooks/pitchtwin",
    }),
  }); // → 202 { id, status: "queued", … }
}

The response is 202 with status: "queued". When rendering finishes, PitchTwin POSTs a signed generation.succeeded event to your webhook_url, or you poll GET /generations/:id. The result includes video_url (MP4), audio_url, duration_ms, tokens_charged and words. The webhook side is covered in Async video generation with webhooks.

Why send the alignment at all

Alignment is optional. The video renders the same without it. What you lose is the words array: in audio mode it is built from your alignment, so without one it comes back empty. Word timings are what you need for captions, word highlighting and search inside videos, and Captions from word timestamps shows how to turn them into WebVTT or SRT.

If you send audio from somewhere other than ElevenLabs, PitchTwin also accepts a simpler shape: { characters, start_times, end_times }, with times in seconds and one entry per character, spaces included.

Long scripts and multiple lines

Audio mode takes up to 10 minutes of audio per generation. For longer content, or for scripts you assemble from several parts, you have two options:

  1. One generation per section. Simple, and each section can be re-rendered on its own when the script changes. This suits training modules and onboarding flows.
  2. One audio file for the whole video. If you synthesize in chunks and concatenate the audio yourself, you also have to merge the alignments: shift each chunk's start and end times by the total duration of the chunks before it. ElevenLabs' previous_text and next_text request fields give each chunk context from its neighbours, which helps intonation stay consistent across the joins.

Either way, keep the avatar the same across sections. Each avatar is pinned to its engine version, so a video rendered next month matches one rendered today.

Languages

Because the voice comes from your audio, the avatar speaks whatever language ElevenLabs produced. That makes audio mode a natural fit for localization: one avatar, many voiced tracks. Multilingual AI avatar videos: TTS vs dubbed audio compares this to letting PitchTwin voice each language.

Practical notes

  • Consent. PitchTwin's Terms require every person in an avatar to have agreed to it. If the ElevenLabs voice is a clone of a real person, get their consent for the voice as well.
  • Keys. Keep both the ElevenLabs and PitchTwin keys on your server.
  • Errors. An alignment whose arrays differ in length or run past the end of the audio is rejected with invalid_alignment. A 402 insufficient_tokens means the account needs a top-up. Audio mode is charged up front for the exact audio length, and failed generations are refunded in full automatically.
  • Signed links. video_url and audio_url expire after 24 hours, so copy the MP4 to your own storage.

Try it

Sign up for 60 free tokens (no card), create an avatar from a short clip, and run renderLine with one of your ElevenLabs voices. The API docs cover the audio mode fields in full.

Keep reading