Whisper Large v3 Turbo Mn LoRA

Whisper Large v3 Turbo — Mongolian (fine-tuned)

A Mongolian automatic speech recognition (ASR) model, fine-tuned from openai/whisper-large-v3-turbo with LoRA on a mixed Mongolian speech corpus. The base model has almost no usable Mongolian ability out of the box; this fine-tune makes it practical for Mongolian (Cyrillic) transcription.

This repository contains a standalone merged model (the LoRA adapter has been folded into the base weights) — use it like any Whisper model, no PEFT required.

Current version: 2 epochs. This model was updated from a 1-epoch checkpoint to a 2-epoch checkpoint, which lowered validation loss from 0.330 → 0.283 and WER from 35.5% → 27.4%. See the changelog at the bottom.

Results

Base vs. fine-tuned, greedy decoding, on a fixed random 200-clip subset of the held-out validation set (Common Voice–dominant). Same clips across versions, so the numbers are directly comparable. Lower is better.

Model	WER	CER
openai/whisper-large-v3-turbo (base)	98.2%	81.2%
this model — 1 epoch (previous)	35.5%	14.2%
this model — 2 epochs (current)	27.4%	11.0%

⚠️ These figures are from a 200-clip quick evaluation, not the full validation set — treat them as indicative (±a few points). The final training eval loss was 0.2828 and was still decreasing at the end of the run, so the model was not yet fully saturated — further training could help.

The base model frequently produces phonetic garbage or drifts into other scripts for Mongolian audio; the fine-tune produces correct Cyrillic Mongolian and is often word-perfect on clean, read-speech clips.

Usage

import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="Blgn94/whisper-large-v3-turbo-mn-lora",
    torch_dtype=torch.float16,
    device="cuda",  # or "cpu"
)

result = pipe("audio.mp3", generate_kwargs={"language": "mongolian", "task": "transcribe"})
print(result["text"])

Or with the model/processor directly:

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import librosa

model_id = "Blgn94/whisper-large-v3-turbo-mn-lora"
processor = WhisperProcessor.from_pretrained(model_id, language="mongolian", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda").eval()

audio, _ = librosa.load("audio.mp3", sr=16000, mono=True)
feats = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt").input_features
feats = feats.to("cuda", torch.float16)
with torch.no_grad():
    ids = model.generate(feats, language="mongolian", task="transcribe", max_new_tokens=225)
print(processor.tokenizer.batch_decode(ids, skip_special_tokens=True)[0].strip())

Audio must be 16 kHz mono. The Mongolian transcription defaults are baked into the model's generation_config, so language/task are optional but shown for clarity.

Training

• Base model: openai/whisper-large-v3-turbo (809M params; full large-v3 encoder + a 4-layer decoder)
• Method: LoRA (PEFT) on an 8-bit-quantized frozen base
  • rank r=32, alpha=64, dropout 0.05
  • target modules: q_proj, v_proj
  • trainable params: 6.55M (0.80% of the 815M total)
• Schedule: 2 epochs (7,166 steps), effective batch size 16 (batch 2 × grad-accum 8), learning rate 1e-3 with 50 warmup steps, linear decay, fp16, gradient checkpointing. Epoch 2 was a resumed run: the LR schedule was rebuilt over the 2-epoch total, giving a warm restart (~5e-4) that decayed back to 0.
• Hardware: single NVIDIA RTX 5060 (8 GB), ~7 hours per epoch (~14 h total)

Validation loss by step

Step	Epoch	eval loss
1,000	1	0.579
2,000	1	0.451
3,000	1	0.372
3,583	1 (end)	0.330
4,000	2	0.408 ← warm-restart bump
5,000	2	0.360
6,000	2	0.320
7,000	2	0.286
7,166	2 (end)	0.283

Training data

~57k Mongolian audio–transcript pairs, mixed sources:

Source	Share	Description
Common Voice	~86%	Crowdsourced read sentences
FLEURS	~7%	Read Wikipedia sentences
MBSpeech	~6%	Read audiobook (Bible) corpus
YouTube	~1%	Scraped + segmented Mongolian audio

Limitations

• Read-speech bias. The training mix is dominated by clean, read Common Voice audio. Expect higher error rates on spontaneous / conversational speech, noisy environments, strong dialects, and telephony-band (8 kHz) audio, none of which are well represented.
• Cyrillic only. Trained on Cyrillic Mongolian transcripts.
• Not fully saturated. Eval loss was still dropping at the end of epoch 2, so additional epochs and/or domain-matched data would likely improve it further.

Changelog

• v2 (2 epochs) — eval loss 0.283, WER 27.4% / CER 11.0% (200-clip subset).
• v1 (1 epoch) — eval loss 0.330, WER 35.5% / CER 14.2%.

License

Inherits Apache-2.0 from the base openai/whisper-large-v3-turbo.