Your customers want the same avatar video in Spanish, German and Japanese, and they want the presenter to look and sound like the same person in every version. The video side is the easy part. The hard decisions are about the audio: who generates the speech, who checks the translation, and how much control you need over the result.
This guide compares the two ways to make multilingual AI avatar videos with an avatar API, built-in text-to-speech and bring-your-own dubbed audio, and shows when each one fits.
The two approaches
With PitchTwin, every generation uses exactly one of three inputs: text, script, or audio. That gives you two modes:
- Text mode (TTS). You send translated text or a
scriptarray of segments with pauses. PitchTwin synthesizes speech in the avatar's cloned voice and generates the video. You can setvoice.languageto an ISO 639-1 code such as"de". - Audio mode (dubbed audio). You send finished audio: from your own TTS provider, a dubbing service or a human voice actor. Optionally include character-level alignment. PitchTwin generates the video to match that exact audio.
In both modes the video comes from the same avatar, which is built from real video of the person and keeps that footage. The face stays 1:1 like the person across every language because it is the same source footage each time.
Side-by-side comparison
| Text mode (built-in TTS) | Audio mode (your audio) | |
|---|---|---|
| Input | text or script | audio: { url, alignment? } |
| Voice | Avatar's cloned voice | Whatever you provide |
| Requires a voice on the avatar | Yes | No (has_voice: false works) |
| Price | 1.5 tokens per second | 1 token per second, plus your audio provider |
| Control over delivery | Speed (0.7–1.2) and pauses | Full: any voice, any edit |
| Integration effort | One API call | Your audio pipeline plus one call |
| Best for | Short, templated, high-volume messages | Reviewed content, brand voices, long-form |
Neither is better in general. The right default depends on how much the audio matters to your customer and whether they already have an audio pipeline.
When text mode fits
Text mode is the shortest path from a translated string to a finished video. It fits when:
- Messages are short and templated, like onboarding greetings, outreach or notifications, where you generate many variations.
- You want the presenter's own voice in every language without managing a second vendor.
- Your product generates content on the fly and nobody listens to each file before it ships.
A German request looks like this:
{
"avatar_id": "av_3f9c1e7a2b4d6e80",
"script": [
{ "text": "Hallo Anna, willkommen bei Northwind.", "pause_after_ms": 400 },
{ "text": "In diesem kurzen Video zeige ich dir die ersten drei Schritte." }
],
"voice": { "speed": 1.0, "language": "de" },
"webhook_url": "https://app.example.com/webhooks/pitchtwin"
}
Setting voice.language is optional; the docs recommend it only when automatic detection picks the wrong language, which can happen with short texts or texts full of product names. The voice comes from the avatar's clone, made from at least 10 seconds of speech in the source clip or from a separate voice sample (1–2 minutes gives the best clone). Test each target language with your real avatars in the dashboard Playground before you offer it to customers, because how a cloned voice carries into another language varies by speaker.
When audio mode fits
Audio mode hands you full control. It fits when:
- The audio gets reviewed. Training, compliance and marketing content usually goes through a native speaker. Generate the audio, get it approved, then generate the video from that exact file.
- You already have a voice provider. If your product uses ElevenLabs or another TTS vendor, keep that contract and voice library. ElevenLabs lists language coverage per model: Eleven v3 at 70+ languages, Multilingual v2 at 29 and Flash v2.5 at 32 (ElevenLabs models).
- You use human voice actors or a dubbing service. Any finished track works, up to 10 minutes per generation.
- The avatar has no voice clone. Some customers want a different narrator for each language, or the person did not provide a voice sample.
The request is short:
{
"avatar_id": "av_3f9c1e7a2b4d6e80",
"audio": {
"url": "https://cdn.example.com/approved/lesson-3-es.mp3",
"alignment": {
"characters": ["H", "o", "l", "a"],
"character_start_times_seconds": [0.0, 0.08, 0.15, 0.21],
"character_end_times_seconds": [0.08, 0.15, 0.21, 0.3]
}
}
}
Alignment is optional. If your provider returns character timings, as the ElevenLabs with-timestamps endpoint does, pass them as is. For a full walkthrough with code, see using your own ElevenLabs audio with an avatar API.
Problems that show up in every language
Whichever mode you choose, the same issues come up once you go past English.
Translation quality
Neither mode translates for you. Whether you use machine translation or a human, have a native speaker review templates before they go into production. Errors in a video are more visible and harder to fix than errors in an email.
Names, numbers and brand terms
Product names, prices and dates are where speech goes wrong. Write numbers out the way they should be spoken in each language, and keep a per-language list of brand terms with spelling hints. In text mode, test any template that includes a customer's name in several languages.
Length and cost
The same message usually runs to a different length in each language, and video length drives cost. Billing is per second of output, so budget per language from real generations rather than from the English version. The duration_ms and tokens_charged fields on each generation give you exact numbers to plan with.
Captions
Every generation returns words with word-level timings in seconds, in both modes. Use them to build subtitles per language instead of running a separate transcription step. The approach is covered in captions for avatar videos from word timestamps.
A practical default
For most B2B products, offer both:
- Use text mode for templated, high-volume messages where the presenter's own voice matters and speed of integration wins.
- Use audio mode for content that a human approves, or when the customer brings their own voice provider.
- Store the language and mode on each generation record so you can report cost per language and spot templates that run long.
Because each avatar is pinned to its engine version, the presenter looks the same in a video you generate today and one you generate next quarter, in any language. The multilingual avatar video page summarizes the setup, and pricing has the per-minute numbers for both modes.
Try it
Sign up at /signup for 60 free tokens with no card. Create an avatar, generate the same message in text mode and in audio mode, and compare them before deciding which to build first.