Arabic Tts Xtts v2

Arabic TTS: Improving XTTS-v2 for Arabic Speech Synthesis

A systematic project to improve Coqui XTTS-v2 for high-quality Arabic text-to-speech generation. This repository documents the full journey from baseline evaluation to fine-tuned deployment, with reproducible benchmarks at every stage.

Pipeline Animation

Table of Contents

• Project Overview
• Where This Project Sits in AI
• Architecture
• Setup & Installation
• Phase 1: Baseline Evaluation
• Phase 2: Compatibility Fixes
• Phase 3: Arabic Text Preprocessing
• Phase 4: Inference Pipeline Improvements
• Phase 5: Data Collection & Preparation
• Phase 6: Fine-Tuning
• Phase 7: Evaluation & Comparison
• Phase 8: Studio-Quality Audio Upsampling
• Resource Usage
• Results
• Known Limitations & Future Improvements
• Industry Use Cases
• Usage
• License

---

Project Overview

XTTS-v2 is a powerful multilingual TTS model supporting 17 languages out of the box — a remarkable achievement by the Coqui team. This project builds on that foundation to push Arabic output quality further by:

1. Expanding Arabic training data — supplementing the base model with additional high-quality Arabic speech data
2. Adding Arabic text preprocessing — introducing diacritization (tashkeel) and hamza normalization to help the model with Arabic-specific orthography
3. Optimizing the tokenizer for Arabic — adapting the shared multilingual tokenizer to better handle Arabic text patterns
4. Tuning inference for Arabic prosody — adjusting generation parameters to better match Arabic speech rhythms and intonation
5. Building a post-processing pipeline — adding pause control, silence management, and output refinement tailored to Arabic

This project addresses each area systematically, with before/after comparisons at every stage.

Goals

• Natural-sounding Arabic speech — produce voices that sound authentically human, not robotic or accented
• Multi-dialect support — generate speech across different Arabic dialects and accents (e.g. Sudanese, Egyptian, Gulf, Levantine, North African, MSA) by fine-tuning on dialect-specific speakers
• Document a reproducible pipeline from base model to production-quality Arabic TTS
• Provide quantitative benchmarks (MOS scores, RTF, SNR) at each improvement stage
• Release fine-tuned checkpoints and preprocessing tools for the Arabic TTS community

---

Where This Project Sits in AI

AI/ML/DL/Data Science Domains

Figure 1: The relationship between AI, Machine Learning, Deep Learning, and Data Science — and where each part of this project applies.

This project spans all four domains. Here is where each phase of our work falls:

Domain	What we did in this project	Phase
Artificial Intelligence	End-to-end system that converts Arabic text to natural-sounding speech — a core AI application	All phases
Deep Learning	Fine-tuned a 370M-parameter GPT-2 transformer + HiFiGAN vocoder on Arabic speech data	Phase 6 (Training)
Machine Learning	Speaker clustering with ECAPA-TDNN embeddings + Agglomerative Clustering to identify speakers in unlabeled data	Phase 5 (Data Preparation)
Data Science	Dataset evaluation, quality filtering, statistical analysis, SNR measurement, Whisper-based alignment verification, visualization of results	Phases 5, 7 (Data & Evaluation)

In detail:

• Deep Learning powers the core model — the GPT-2 backbone generates audio codes autoregressively, the HiFiGAN vocoder converts them to waveforms, and the Perceiver Resampler handles speaker conditioning. Fine-tuning these neural networks on Arabic data is a Deep Learning task
• Machine Learning solved our speaker identification problem — with 103K unlabeled clips, we used ECAPA-TDNN (a neural feature extractor) combined with Agglomerative Clustering (a classical ML algorithm) to find and extract 5,000 clips from the same speaker
• Data Science drove every decision about data quality — from evaluating 6 candidate datasets, to measuring SNR and silence ratios, to building the 3-layer sanity check pipeline, to generating comparison charts and metrics
• AI is the umbrella — the final system takes Arabic text as input and produces human-like speech as output, combining all three disciplines into a practical application

---

Architecture

XTTS-v2 Model Components

XTTS-v2 Architecture Figure 2: XTTS-v2 architecture — text and reference audio are processed through separate pathways, combined in the GPT-2 transformer, and decoded to a 24kHz waveform.

Key Specifications

Component	Details
GPT Backbone	30 layers, 1024 hidden dim, 16 heads, ~350M params
Tokenizer	VoiceBPE, 6,681 tokens, 17 languages
Audio Codec	Discrete VAE, 1,026 codebook entries
Vocoder	HiFiGAN, [8,8,2,2] upsampling, ResBlock1
Speaker Encoder	ResNet50 + SE blocks, 512-dim output
Conditioning	Perceiver Resampler, 2 layers, 32 latents
Output	24,000 Hz WAV
Input Sample Rate	22,050 Hz
Max Text Tokens	402
Max Audio Tokens	605
Character Limit	166 per chunk (Arabic)
Total Parameters	~370M

---

Setup & Installation

Hardware

This project was developed on an NVIDIA DGX Spark — a desktop AI workstation with a Grace Blackwell architecture. If you're using different hardware, you may need to adjust CUDA versions, PyTorch builds, and batch sizes accordingly.

Spec	DGX Spark (this project)
GPU	NVIDIA GB10 (Blackwell)
GPU Memory	121.7 GB (unified)
CPU Architecture	ARM (aarch64)
System RAM	128 GB
OS	Ubuntu 24.04 LTS
CUDA	13.0
NVIDIA Driver	580.142

[!NOTE]

Running on different hardware?

Consumer GPUs (RTX 3090, 4090, etc.):

• You'll need PyTorch built for your CUDA version (e.g., cu118, cu121, cu124 instead of cu130)
• With 24GB VRAM, reduce batch_size to 2 and grad_accumulation_steps to 1 during fine-tuning
• Inference works fine on 8GB+ VRAM

Data Center GPUs (A100, H100):

• Should work out of the box with matching CUDA/PyTorch versions
• Can increase batch sizes for faster training

x86 vs ARM (aarch64):

• DGX Spark uses ARM. If you're on x86 (most desktops/servers), install the x86 PyTorch build — the code is the same, only the binary packages differ
• pip install TTS will automatically pull the correct architecture

CPU only:

• Inference is possible but very slow (~10x slower than GPU)
• Fine-tuning on CPU is not practical

Key: Match your PyTorch version to your CUDA version. Check compatibility at pytorch.org/get-started.

Software Environment

Package	Version	Purpose
Python	3.10.20	Runtime (via Miniconda)
PyTorch	2.11.0+cu130	Deep learning framework — match to your CUDA version
CUDA Toolkit	13.0	Match to your GPU driver
Coqui TTS	0.22.0	Core XTTS-v2 framework
Transformers	4.44.2	Pinned — v5.x breaks Coqui TTS (see Phase 2)
SpeechBrain	1.1.0	ECAPA-TDNN speaker embeddings
OpenAI Whisper	latest	Alignment verification (large-v3 model)
AudioSR	0.0.7	Neural audio upsampling (24→48 kHz)
Mishkal	0.4.1	Arabic tashkeel — optional, for formal MSA mode
PyArabic	0.6.15	Arabic text utilities
datasets	4.8.4	HuggingFace dataset loading
torchcodec	0.11.1	Audio decoding for HuggingFace datasets
matplotlib	—	Chart generation
umap-learn	—	Dimensionality reduction for speaker cluster visualization

Installation Steps

# 1. Create conda environment
conda create -n new-arabic-tts python=3.10 -y
conda activate new-arabic-tts

# 2. Install Coqui TTS
pip install TTS

# 3. Pin transformers for compatibility (see Phase 2)
pip install transformers==4.44.2

# 4. Download XTTS-v2 base model
echo "y" | python -c "
from TTS.utils.manage import ModelManager
mm = ModelManager()
mm.download_model('tts_models/multilingual/multi-dataset/xtts_v2')
"

# 5. Copy model to project directory
mkdir -p models/base
cp -r ~/.local/share/tts/tts_models--multilingual--multi-dataset--xtts_v2/* models/base/

Project Structure

New Arabic TTS/
├── README.md                       # This file
├── CHANGELOG.md                    # Detailed change log
├── configs/                        # Configuration files
├── data/
│   └── egyptian/                   # Phase 5: Prepared training data
│       ├── metadata_train.csv      #   Training split (cleaned)
│       ├── metadata_eval.csv       #   Evaluation split (cleaned)
│       ├── wavs/                   #   22,050 Hz mono WAV files
│       └── rejected/              #   Clips that failed sanity check
├── docs/
│   ├── DATA_FORMAT.md              # Data format guide for fine-tuning
│   ├── images/
│   │   ├── speaker_clusters_umap.png   # Phase 5: UMAP speaker clusters
│   │   ├── cluster_sizes.png           # Phase 5: Cluster size chart
│   │   ├── similarity_distribution.png # Phase 5: Speaker consistency cutoff
│   │   ├── duration_distribution.png   # Phase 5: Duration histogram
│   │   ├── comparison_waveforms.png    # Phase 7: Waveform comparison
│   │   ├── comparison_spectrogram.png  # Phase 7: Spectrogram comparison
│   │   ├── comparison_metrics.png      # Phase 7: Metrics bar chart
│   │   └── training_loss.png           # Phase 7: Training loss curve
│   └── benchmarks/
│       ├── baseline.json               # Phase 1: Baseline metrics
│       ├── dataset_preparation.json    # Phase 5: Pipeline report
│       ├── sanity_check.json           # Phase 5: Data quality report
│       ├── training_summary.json       # Phase 6: Training results
│       └── evaluation.json             # Phase 7: Comparison metrics
├── models/
│   └── base/                       # XTTS-v2 base model (1.8GB)
│       ├── config.json
│       ├── model.pth
│       ├── speakers_xtts.pth       # 58 built-in speakers
│       └── vocab.json
├── outputs/
│   ├── original model/             # Phase 1: Raw base model outputs
│   │   └── Original_Model_test.wav #   Baseline benchmark (standard demo text)
│   ├── improved model/             # Phase 3+4: Text preprocessing + post-processing
│   │   ├── Improved_Model_test.wav #   Same text, improved pipeline
│   │   └── Improved_Model_test_stats.json  # Per-chunk generation stats
│   └── finetuned model/             # Phase 6: After fine-tuning
│       ├── Finetuned_Model_test.wav          # Fine-tuned output (same demo text)
│       └── Finetuned_Model_test_stats.json   # Generation stats
└── scripts/
    ├── arabic_preprocessor.py      # Phase 3: Arabic text preprocessing
    ├── baseline_test.py            # Phase 1: Baseline benchmark generator
    ├── infer.py                    # Phase 4: Improved inference...