Speech-to-Creative Pipeline: speech recognition + language swap + lip-sync
One video creative is scaled to 5+ languages and countries automatically. Whisper transcribes speech → GPT-4o translates and rewrites for the market and persona → ElevenLabs clones the voice with the correct accent → wav2lip syncs lip movement to the new audio. What used to require a recording studio and a week of work — now happens in an evening.
Competitor research exists, but it doesn't scale by hand
In performance marketing, ad intelligence services (AdHeart, AdSpy, AdsLibrary, Anstrex) show thousands of competitors' working ads — but you can only borrow what does not trigger a copyright strike. Narrator voice, background, actor's face, lip-sync — everything covered by DMCA requires recreation.
The workflow used to look like this: watch a competitor's video → write the transcript by hand → hand it to a copywriter to rewrite for your own offer → hire voice actor via Fiverr → wait 1-3 days → edit in Final Cut → iterate. 4-6 hours of work per variation, and that's without lip-sync.
With budgets of $2-5k/day on A/B tests, how fast you can produce is the competitive advantage. When you have 8-10 funnels running at once and each needs 20-30 fresh creatives per week — the manual workflow stops working.
Four stages: speech-to-text → language model → speech synthesis → lip-sync
Whisper transcription of competitor videos
whisper-large-v3 or OpenAI whisper-1 via API. Supports Russian, English, Balkan, Turkic, Spanish. Outputs SRT with timecodes — we know exactly which phrase is on which second.
GPT — translation, localization, persona rewrite
GPT-4o does three tasks in one pass: (a) breaks the transcript into compositional blocks — the hook (first 3 seconds), the pain, the proof, the call to action; (b) translates into the target language (Russian / English / Serbian / Polish / Turkish) with cultural nuance — not literal but as "a native speaker would say it"; (c) rewrites it for the target offer, audience and country, preserving the rhythm and emotional triggers of the original.
Few-shot prompts with ready "before → after" examples per language and niche. The main focus is preserving timecodes: each phrase's duration must match the original or lip-sync will break. Output — 10-30 variants × 5+ languages, ranked by predicted engagement.
ElevenLabs — voice cloning with the right accent
ElevenLabs Multilingual v2 supports 29 languages in a single model — the cloned voice sounds in each with the correct accent. This matters: if the original speaker is an American with a Southern accent, her clone in Serbian won't sound robotic but like a natural native Serbian speaker.
Two strategies: (a) voice cloning — a 30-second sample is enough for a high-quality clone; (b) stock voices for fast A/B by timbre (male/female, age, emotional tone). Stability / similarity / style settings are tuned per niche. Output — audio.wav matching the original's duration (important for the next step).
wav2lip — lip movement synchronization
At this stage the "magic" appears — the actress in the video starts speaking in the new language as if it was re-shot. wav2lip analyzes the source video + new audio.wav and redraws the mouth region frame by frame so the lip movement matches the new speech. A GPU is needed, but it's hours of compute, not days at a studio.
Simple case (voice-over / off-screen): ffmpeg just replaces the audio track. Complex case (talking head): wav2lip or SadTalker for face sync. Output — finished mp4 for ad platforms (FB Ads / TikTok / VK Ads / Yandex Direct).
Technologies and infrastructure
- OpenAI Whisper API (whisper-1)
- whisper-large-v3 (self-hosted, at large volumes)
- SRT parsing for timecodes
- GPT-4o / Claude Sonnet 4.5
- few-shot prompts with hook → rewrite examples
- structured outputs (JSON schema)
- ElevenLabs Multilingual v2
- Voice cloning (30-second sample is enough)
- Voice style settings: stability / similarity / style
- ffmpeg for audio-replace (voice-over)
- wav2lip / SadTalker for talking head
- Python orchestration + task queue
What changes in the funnel
per creative variation (audio-replace, no lip-sync)
variants × languages (1 actress → 5+ countries without a re-shoot)
Whisper + GPT + ElevenLabs (at API rates, no lip-sync)
The main effect is iteration speed. An A/B test of 20 opening hooks instead of 2 launches in an evening, not a week. The winning combinations show up in the first 24-48 hours; the losers are switched off before they burn budget. Cost per install (CPI) drops 15-30% because the opening matches the audience better.
Where it works, where it doesn't
- · Performance agencies with an in-house creative team of 5+
- · Performance teams and online retailers producing a lot of video
- · In-house e-commerce marketing that publishes user-made video regularly
- · Launches in 5+ countries simultaneously (multi-language from one master script)
- · Podcasters and online-course businesses cutting short-form clips
- · Direct copying of other people's creatives (DMCA / Copyright Strike)
- · Regulated niches (medicine, pharma, finance) — need media lawyers for compliance
- · Voice cloning without consent of the voice owner (banned by EU AI Act)
- · Long-form (10+ minute videos) — TTS cost and editing time comparable to an expensive human actor
Ethical note: the pipeline is intended for scaling your own ideas and original scripts. Using others' videos and voices without permission violates Copyright and the AI Act. I transcribe other people's ads as research to understand the market, then write my own script, my own audio, and my own video.
Related cases
GMBSPYLAB — collecting the public FB Ad Library into my own Telegram feed
Async Python and Postgres 16: 30–75k raw cards a day, perceptual dedup leaves 5–12k unique ones, arq queues…
End-to-end attribution: from ad click to order
A stdlib middleware takes Shopify webhooks, verifies HMAC, fires an S2S postback to the tracker and a…
Paid Shopify orders → warehouse spreadsheets
Orders, tracking numbers, statuses and cancellations land by themselves every 30 minutes. An Advanced Sheets…
The 5,000 ₽ audit — with a concrete report and quote
I'll tell you what to deploy in your business first, what the payback looks like, and whether you need AI for the task at all (sometimes you don't).
Or just send your question — I reply within 2 hours