Qwen3 Asr 0.6b Arabic Dialectal

LLM-ASR (multimodal). Full fine-tune, targets stripped of tashkil + tags (trial t3). 3 epochs. Needs transformers>=5.14 for qwen3_asr.

Model comparison — all Arabic ASR models

Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags, keep punctuation + dialect spelling) — so every row is directly comparable.

• Best fine-tuned: whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic a close second (0.357).
• Best zero-shot: cohere-transcribe-arabic (0.457, Arabic-specialized). A full fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning with best-checkpoint selection was the fix.
• Streaming / low-latency: nemotron-3.5-asr.
• Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining; cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).

Dataset

Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).

• ~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
• Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi (Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
• Augmented: each clean clip is expanded with variants (noise, music, speed, voice, reverb/codec) via an augmentation column. Derived from oddadmix/dialectal-arabic-lahgtna-v2-smaller.

Preprocessing (applied to targets)

• Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags ([laughter], [exhale], [inhale], [mumble], [cough], timestamps, ...).
• Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and sentence punctuation. Output is undiacritized.
• Filtered: dropped clips > 30 s (~10%; Whisper's encoder limit) and a few corrupt rows (12k-token transcripts). ~62% of rows had harakat, ~28% had non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.

Training

• Base: Qwen/Qwen3-ASR-0.6B-hf · full fine-tune (no LoRA)
• Effective batch 16 · LR 1e-5 (warmup 500, cosine) · 6903 steps · best @ step None · ~102 min
• Hardware: a single 32 GB GPU, bf16 autocast (fp32 weights + generate)
• Metric: WER/CER on cleaned references (clean-text normalized)

Fine-tuning (reproduce)

The exact fine-tuning code is bundled in this repo (train_qwen.py, normalize.py, eval_hf_asr.py) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).

pip install -r requirements.txt        # recent transformers (qwen3_asr) + peft

python train_qwen.py --model_id Qwen/Qwen3-ASR-0.6B-hf \
  --dataset_name oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented --audio_column audio --text_column text \
  --eval_split test --language Arabic --strip_tashkil --strip_tags \
  --output_dir my-run

python eval_hf_asr.py --model_type qwen --model my-run   # WER / CER

Note: Qwen3-ASR's chat template drops the transcript during training — train_qwen.py builds the target manually with the prompt masked to -100 (see its DataCollator).

Usage

# needs transformers>=5.14 (ships the qwen3_asr architecture)
import torch, torchaudio
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

repo = "oddadmix/qwen3-asr-0.6b-arabic-dialectal"
proc = AutoProcessor.from_pretrained(repo)
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16).to("cuda").eval()

wav, sr = torchaudio.load("clip.wav")          # 16 kHz mono
inp = proc.apply_transcription_request(audio=wav.mean(0).numpy(),
        language="Arabic").to(model.device, model.dtype)
ids = model.generate(**inp, max_new_tokens=256)
print(proc.batch_decode(ids, skip_special_tokens=True)[0])

Learnings & notes

• Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags (rather than aggressive letter-folding) was key for a multi-dialect model.
• Whisper fine-tunes furthest; the offline (full-context) models beat the streaming Nemotron on pure WER, though Nemotron has the best zero-shot (native Arabic) and offers streaming.
• Blackwell (sm_120) GPU gotchas for the NeMo/Nemotron model: numba's RNNT loss needs numba==0.61.2 + NUMBA_CUDA_USE_NVIDIA_BINDING=1 + cuda-python<13 to emit sm_120 code, and fp32 (the numba loss can't take bf16). Nemotron 3.5 requires NeMo main (its EncDecRNNTBPEModelWithPrompt class isn't in any release) and a per-sample lang=ar prompt.

Limitations

• Trained on augmented, partly synthetic multi-dialect data; real-world dialect coverage varies (Maghrebi is the hardest).
• Output is undiacritized and lower-cased for Latin tokens.
• Offline model (≤30 s chunks); not streaming.