Localizing video used to mean voice actors, a studio and weeks of turnaround. Dubbing APIs have made the audio side cheap and fast, but they leave a visible problem: the speaker's mouth still moves for the original language. For talking-head content, viewers notice.
If you are building a dubbing or localization tool, an avatar API can close that gap by generating a new video of the same speaker delivering the translated audio. This post covers where that helps, where it does not, and a pipeline you can run today.
Two different jobs
It helps to separate two things that both get called "dubbing":
- Dubbing an existing video. You keep the original edit, b-roll, screen recordings and music, and replace the speech track. The output is the same video with a new language.
- Generating a new presenter video. For segments where a person talks to camera, you generate a new video of that person speaking the translated audio.
Dedicated dubbing services do the first job well. ElevenLabs Dubbing, for example, translates audio and video while keeping each speaker's voice characteristics and the background audio, and its docs list support for 90+ languages (ElevenLabs Dubbing docs). If your content is mostly voiceover, screen recordings, or speakers who are small in the frame, dubbed audio alone may be all you need.
An avatar API does the second job. PitchTwin generates a new MP4 of the person from their avatar, driven by the audio you provide. It does not edit your existing long-form video. You use it for the talking-head parts and cut them back into your edit.
When a new presenter video is worth it
A presenter video costs extra tokens and adds a step, so be specific about where it pays off:
| Content | Dubbed audio only | New presenter video |
|---|---|---|
| Screen recording with voiceover | Usually enough | Not needed |
| Course intro, presenter full-frame | Mouth mismatch is visible | Good fit |
| Sales or onboarding message to camera | Mouth mismatch is visible | Good fit |
| Interview with several people, wide shot | Usually acceptable | Hard; one avatar per person |
| Content with heavy camera movement or b-roll | Usually enough | Only for the to-camera shots |
The strongest fit is short, frequent, to-camera content: a training platform's lesson intros, a SaaS onboarding message, a founder update sent in five languages.
How it works with PitchTwin
The presenter needs an avatar, built from a 10–120 second clip of them on camera. PitchTwin builds the avatar from that real video and keeps the footage, so the localized version still looks 1:1 like the person: same skin, hair, lighting and expressions. More on that on the multilingual avatar video page.
Then you have two ways to get speech into the video:
- Audio mode. You bring finished audio: the output of a dubbing service, a TTS call, or a human voice actor. You send
audio: { url, alignment? }. This costs 1 token per second of video. - Text mode. You send the translated text or a
scriptarray, and PitchTwin speaks it in the avatar's cloned voice. You can passvoice.languageas an ISO 639-1 code. This costs 1.5 tokens per second.
For a dubbing product, audio mode is usually the right default. You already have a translation and voice pipeline, and you want the avatar video to match the exact audio your customer approved. The trade-offs between the two modes are covered in multilingual AI avatar videos: TTS vs dubbed audio.
A pipeline in Python
This example takes a translated script, generates speech with character timestamps from ElevenLabs, and sends the audio plus alignment to PitchTwin. The ElevenLabs with-timestamps endpoint returns base64 audio and an alignment object with characters, character_start_times_seconds and character_end_times_seconds (ElevenLabs API reference). PitchTwin accepts that alignment format as is.
import base64
import os
import requests
PT = "https://pitchtwin.ai/api/v1"
PT_HEADERS = {"Authorization": f"Bearer {os.environ['PITCHTWIN_API_KEY']}"}
def eleven_tts(text: str, voice_id: str) -> tuple[bytes, dict]:
r = requests.post(
f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}/with-timestamps",
headers={"xi-api-key": os.environ["ELEVENLABS_API_KEY"]},
json={"text": text, "model_id": "eleven_multilingual_v2"},
timeout=120,
)
r.raise_for_status()
body = r.json()
return base64.b64decode(body["audio_base64"]), body["alignment"]
def upload_to_pitchtwin(data: bytes) -> str:
r = requests.post(f"{PT}/uploads", headers=PT_HEADERS, timeout=30)
r.raise_for_status()
up = r.json()
requests.put(up["upload_url"], data=data, timeout=120).raise_for_status()
return up["ref"] # "upload:<file_id>"
def localize(avatar_id: str, text: str, voice_id: str) -> str:
audio, alignment = eleven_tts(text, voice_id)
ref = upload_to_pitchtwin(audio)
r = requests.post(
f"{PT}/generations",
headers=PT_HEADERS,
json={
"avatar_id": avatar_id,
"audio": {"url": ref, "alignment": alignment},
"webhook_url": "https://app.example.com/webhooks/pitchtwin",
},
timeout=30,
)
r.raise_for_status()
return r.json()["id"] # status: "queued"
If you use ElevenLabs Dubbing instead of TTS, the flow is the same: download the dubbed audio for the presenter segment and pass it in audio.url. Alignment is optional; skip it if you do not have character timings.
A few practical notes:
- Audio length. Audio mode accepts up to 10 minutes per generation. Split longer talks at natural breaks and generate each segment.
- File size. Direct uploads go up to 50 MB. For larger files, pass an https URL from your own storage instead.
- Delivery. You receive a signed webhook (
generation.succeededorgeneration.failed) once; pollGET /generations/:idfor anything you missed. The webhooks docs cover signature checks. - Expiry. The returned
video_urlandaudio_urlexpire after 24 hours. Copy the MP4 to your storage before you build the final edit.
Putting the edit back together
PitchTwin outputs an MP4 of the presenter. Your tool owns the timeline: swap the to-camera shots for the localized presenter videos, keep the b-roll and screen recordings with dubbed audio, and mix music underneath. The words array in each generation gives word-level timings in seconds, which you can use for localized subtitles.
Keep the original-language and localized videos tied to the same avatar ID. Each avatar is pinned to its engine version, so a German video you generate next month will match the French one you generated today.
Cost
With your own audio, a 2-minute localized presenter segment uses 120 tokens. On the Growth pack ($39 for 3,000 tokens) that is $0.78 per minute. Your TTS or dubbing provider bills separately. Tokens never expire, failed generations are refunded in full, and there is no watermark or PitchTwin branding on the output, so it fits inside a white-label localization product. See pricing for all packs.
Try it
Sign up at /signup for 60 free tokens with no card, create an avatar from a presenter clip, and send one translated audio file through audio mode. The docs include the full audio and alignment reference.