Use cases

Video Dubbing API: Localizing Avatar Video

How a video dubbing API and an avatar API fit together: when dubbed audio is enough, when to generate the speaker anew, and a Python pipeline.

PitchTwin team, , 5 min read

Localizing video used to mean voice actors, a studio and weeks of turnaround. Dubbing APIs have made the audio side cheap and fast, but they leave a visible problem: the speaker's mouth still moves for the original language. For talking-head content, viewers notice.

If you are building a dubbing or localization tool, an avatar API can close that gap by generating a new video of the same speaker delivering the translated audio. This post covers where that helps, where it does not, and a pipeline you can run today.

Two different jobs

It helps to separate two things that both get called "dubbing":

  • Dubbing an existing video. You keep the original edit, b-roll, screen recordings and music, and replace the speech track. The output is the same video with a new language.
  • Generating a new presenter video. For segments where a person talks to camera, you generate a new video of that person speaking the translated audio.

Dedicated dubbing services do the first job well. ElevenLabs Dubbing, for example, translates audio and video while keeping each speaker's voice characteristics and the background audio, and its docs list support for 90+ languages (ElevenLabs Dubbing docs). If your content is mostly voiceover, screen recordings, or speakers who are small in the frame, dubbed audio alone may be all you need.

An avatar API does the second job. PitchTwin generates a new MP4 of the person from their avatar, driven by the audio you provide. It does not edit your existing long-form video. You use it for the talking-head parts and cut them back into your edit.

When a new presenter video is worth it

A presenter video costs extra tokens and adds a step, so be specific about where it pays off:

ContentDubbed audio onlyNew presenter video
Screen recording with voiceoverUsually enoughNot needed
Course intro, presenter full-frameMouth mismatch is visibleGood fit
Sales or onboarding message to cameraMouth mismatch is visibleGood fit
Interview with several people, wide shotUsually acceptableHard; one avatar per person
Content with heavy camera movement or b-rollUsually enoughOnly for the to-camera shots

The strongest fit is short, frequent, to-camera content: a training platform's lesson intros, a SaaS onboarding message, a founder update sent in five languages.

How it works with PitchTwin

The presenter needs an avatar, built from a 10–120 second clip of them on camera. PitchTwin builds the avatar from that real video and keeps the footage, so the localized version still looks 1:1 like the person: same skin, hair, lighting and expressions. More on that on the multilingual avatar video page.

Then you have two ways to get speech into the video:

  1. Audio mode. You bring finished audio: the output of a dubbing service, a TTS call, or a human voice actor. You send audio: { url, alignment? }. This costs 1 token per second of video.
  2. Text mode. You send the translated text or a script array, and PitchTwin speaks it in the avatar's cloned voice. You can pass voice.language as an ISO 639-1 code. This costs 1.5 tokens per second.

For a dubbing product, audio mode is usually the right default. You already have a translation and voice pipeline, and you want the avatar video to match the exact audio your customer approved. The trade-offs between the two modes are covered in multilingual AI avatar videos: TTS vs dubbed audio.

A pipeline in Python

This example takes a translated script, generates speech with character timestamps from ElevenLabs, and sends the audio plus alignment to PitchTwin. The ElevenLabs with-timestamps endpoint returns base64 audio and an alignment object with characters, character_start_times_seconds and character_end_times_seconds (ElevenLabs API reference). PitchTwin accepts that alignment format as is.

import base64
import os
import requests

PT = "https://pitchtwin.ai/api/v1"
PT_HEADERS = {"Authorization": f"Bearer {os.environ['PITCHTWIN_API_KEY']}"}


def eleven_tts(text: str, voice_id: str) -> tuple[bytes, dict]:
    r = requests.post(
        f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}/with-timestamps",
        headers={"xi-api-key": os.environ["ELEVENLABS_API_KEY"]},
        json={"text": text, "model_id": "eleven_multilingual_v2"},
        timeout=120,
    )
    r.raise_for_status()
    body = r.json()
    return base64.b64decode(body["audio_base64"]), body["alignment"]


def upload_to_pitchtwin(data: bytes) -> str:
    r = requests.post(f"{PT}/uploads", headers=PT_HEADERS, timeout=30)
    r.raise_for_status()
    up = r.json()
    requests.put(up["upload_url"], data=data, timeout=120).raise_for_status()
    return up["ref"]  # "upload:<file_id>"


def localize(avatar_id: str, text: str, voice_id: str) -> str:
    audio, alignment = eleven_tts(text, voice_id)
    ref = upload_to_pitchtwin(audio)
    r = requests.post(
        f"{PT}/generations",
        headers=PT_HEADERS,
        json={
            "avatar_id": avatar_id,
            "audio": {"url": ref, "alignment": alignment},
            "webhook_url": "https://app.example.com/webhooks/pitchtwin",
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["id"]  # status: "queued"

If you use ElevenLabs Dubbing instead of TTS, the flow is the same: download the dubbed audio for the presenter segment and pass it in audio.url. Alignment is optional; skip it if you do not have character timings.

A few practical notes:

  • Audio length. Audio mode accepts up to 10 minutes per generation. Split longer talks at natural breaks and generate each segment.
  • File size. Direct uploads go up to 50 MB. For larger files, pass an https URL from your own storage instead.
  • Delivery. You receive a signed webhook (generation.succeeded or generation.failed) once; poll GET /generations/:id for anything you missed. The webhooks docs cover signature checks.
  • Expiry. The returned video_url and audio_url expire after 24 hours. Copy the MP4 to your storage before you build the final edit.

Putting the edit back together

PitchTwin outputs an MP4 of the presenter. Your tool owns the timeline: swap the to-camera shots for the localized presenter videos, keep the b-roll and screen recordings with dubbed audio, and mix music underneath. The words array in each generation gives word-level timings in seconds, which you can use for localized subtitles.

Keep the original-language and localized videos tied to the same avatar ID. Each avatar is pinned to its engine version, so a German video you generate next month will match the French one you generated today.

Cost

With your own audio, a 2-minute localized presenter segment uses 120 tokens. On the Growth pack ($39 for 3,000 tokens) that is $0.78 per minute. Your TTS or dubbing provider bills separately. Tokens never expire, failed generations are refunded in full, and there is no watermark or PitchTwin branding on the output, so it fits inside a white-label localization product. See pricing for all packs.

Try it

Sign up at /signup for 60 free tokens with no card, create an avatar from a presenter clip, and send one translated audio file through audio mode. The docs include the full audio and alignment reference.

Keep reading