Levantine Codeswitch Tts

--- license: apache-2.0 language: - ar - en tags: - text-to-speech - tts - vits - levantine-arabic - arabic - code-switching - streaming - low-latency - pipecat - real-time base_model: facebook/mms-tts-ara pipeline_tag: text-to-speech library_name: pytorch --- # Levantine ↔ English Code-Switching TTS — Low-Latency & Streaming <p align="center"><b>A research release from Tushe — The Foundry</b><br> <i>Research &amp; Engineering Division, Tushe Development Company — infrastructure and intelligence for resource-constrained, edge-first AI.</i></p> > A streaming text-to-speech system that switches between **Levantine Arabic** and **English**, built for real-time conversational agents on NVIDIA L4 / RTX-3090-class GPUs. It pairs a **deterministic, unit-tested unified-IPA front-end** (which owns all the dialect and code-switching intelligence) with a tiny **non-autoregressive VITS** acoustic model conditioned on a **language-ID embedding**. This card documents the project **exhaustively** — architecture, data, every training stage, ablations, the engineering log of bugs fixed, the measured results, and an unusually candid account of what worked, what didn't, and exactly why. It is written to be reproducible. --- ## 0. TL;DR — what to take away | Production KPI | Target | **Measured (RTX 3090)** | Verdict | |---|---|---|---| | Peak VRAM (inference) | ≤ 3 GB | **193 MB** | ✅ ~16× margin | | Time-to-First-Audio p50 / p95 | < 300 ms | **87 / 144 ms** | ✅ | | Real-Time Factor mean / p95 | < 0.3 | **0.044 / 0.089** | ✅ ~7× margin | | Streaming (chunked) | required | REST + WebSocket + Pipecat | ✅ | | Quality (held-out, n=18) | English | Arabic | Code-switched | Overall | |---|---|---|---|---| | **ASR round-trip CER** ↓ | **0.27** | 0.68 | 0.79 | 0.58 | | ASR round-trip WER ↓ | 0.41 | 1.00 | 0.99 | 0.80 | | **UTMOS** ↑ (1–5) | 2.36 | 2.16 | 2.14 | 2.22 | **The one-paragraph story.** The architecture is validated across the board — it beats *every* latency/VRAM target by a wide margin, and **English is genuinely intelligible** (CER 0.27 — *"I would like to book a flight from Beirut to London"* round-trips through Whisper as *"I would like to ___ the flight from Beirut to…"*). Levantine Arabic is the natural next iteration: this short, single-GPU run used **MSA** audio (the clean diacritized Arabic corpus available), so reaching polished Levantine is a focused *...