Whisper Sinhala Small

🇱🇰 Whisper Sinhala Small

A fine-tuned Whisper-small model for Sinhala (සිංහල) speech recognition. This model significantly improves transcription quality for Sinhala language audio compared to the base openai/whisper-small model, which has no native Sinhala support.

Key Improvements

Metric	Base whisper-small	Fine-tuned	Improvement
WER (Word Error Rate)	130.86%	61.73%	−69.13 pp (45% relative)
CER (Character Error Rate)	244.38%	14.86%	−229.52 pp (94% relative)

Evaluated on 10 validation audio samples from the Sinhala speech corpus.

Why Fine-tuning is Essential

The base Whisper-small model was not trained on Sinhala and produces unusable output — mixing Greek, Telugu, Tamil, Arabic, and Hindi characters with English phrases on Sinhala audio. Fine-tuning transforms this into meaningful Sinhala transcription.

Example (Sinhala song vocals, ~4.7 min):

	Transcript
Base whisper-small	ὁ ὁ ὁ ... వార్ల్ల్ల్ ... I'm running, running, running ... My dear, my dear ... Thank you very much for your time ... (gibberish: Greek, Telugu, English, Arabic, Tamil)
Fine-tuned Sinhala	සුවදට මද නළයි විසික්කල්‍ය සිවදට මද නළයි විසික්කල්‍ය ... සමුදුරු‍රපටු පොල රැලින්‍යේ ... හාමෝයේ වෝපේ ... (readable Sinhala with correct phonetic patterns)

Model Details

Property	Value
Base model	openai/whisper-small
Architecture	WhisperForConditionalGeneration
Parameters	~244M (12 encoder, 12 decoder, d_model=768)
Language	Sinhala (si-LK)
Task	Automatic Speech Recognition (transcription)
Training steps	5,000
Learning rate	1e-4
Batch size	2 (per device), gradient accumulation: 4
Precision	FP32 (for MPS/CUDA compatibility)

Dataset

Fine-tuned on the Sinhala speech corpus (audio_data/si_lk.lines.txt):

• Format: ( audio_id "transcript" ) per line
• Audio: 16 kHz mono WAV files
• Split: 90% training / 10% validation (random seed 42)
• Training samples: ~10,000+
• Validation samples: 10 samples used for WER/CER evaluation

Quick Start

Transcribe audio with the pipeline

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import torch

model_id = "harshanaDisa/whisper-sinhala-small"

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float32

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=device,
    torch_dtype=torch_dtype,
    chunk_length_s=30,
    stride_length_s=5,
)

result = pipe("path/to/audio.mp3")
print(result["text"])

Command-line usage (local)

# Clone repo and transcribe
git clone https://github.com/harshanaDisa/whisper-sinhala.git
cd whisper-sinhala
python transcribe_vocals.py

Training

# Install dependencies
python -m venv .venv && source .venv/bin/activate
pip install torch transformers accelerate soundfile librosa jiwer datasets pandas

# Prepare dataset
python create_samples.py

# Train
python train.py

# Evaluate (fine-tuned only)
python test_model.py

# Compare against base whisper-small
python test_model.py --compare

Limitations

• Domain-specific: Fine-tuned on a specific Sinhala speech corpus; performance may vary on out-of-domain audio (e.g., news, technical content)
• WER still ~62%: The error rate is significantly better than the base model but not perfect — further training or a larger dataset would help
• No timestamps: The model is configured without timestamp tokens (50363) for cleaner transcription output
• Single language: Optimized for Sinhala only; multilingual capabilities of the base Whisper model are degraded

Reproduction

The full training code, dataset preparation scripts, and evaluation pipeline are available in the GitHub repository.

License

This model is released for research and educational purposes. The base Whisper architecture is by OpenAI; the fine-tuned weights are derived from openai/whisper-small.

Citation

@inproceedings{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and others},
  booktitle={ICML},
  year={2023}
}

Links

• HuggingFace Model: harshanaDisa/whisper-sinhala-small
• GitHub Repository: harshanaDisa/whisper-sinhala