Loading
Have a ComfyUI workflow? Get a readable Workflow Doc in your browser.Make a workflow readable.Try it now ↗
A fine-tuned version of OpenAI Whisper Medium for Hindi and Hinglish Automatic Speech Recognition (ASR).
Model source
Source description
A fine-tuned version of OpenAI Whisper Medium for Hindi and Hinglish Automatic Speech Recognition (ASR).
This model was developed to improve Whisper's performance on conversational Hindi and code-mixed Hindi-English speech using a carefully curated synthetic speech corpus.
Sources
1 sourceVerified Sep 22
Model artifacts
1 artifactSource excerpts
2 excerpts| Property | Value |
|---|
| Base Model | openai/whisper-medium |
| Model Type | Whisper Encoder-Decoder Transformer |
| Task | Automatic Speech Recognition |
| Languages | Hindi, Hinglish |
| Framework | Hugging Face Transformers |
OpenAI Whisper provides excellent multilingual speech recognition, but conversational Hindi and Hinglish often remain challenging due to code-mixing, pronunciation variations, and domain-specific vocabulary.
This project fine-tunes Whisper Medium using a curated synthetic dataset to significantly improve recognition quality while preserving Whisper's multilingual capabilities.
The training dataset was generated using a multi-stage synthetic data generation pipeline consisting of:
The focus was on maximizing transcript quality while maintaining linguistic diversity.
Evaluation was performed on an unseen held-out test set.
| Metric | Score |
|---|---|
| Test Samples | 373 |
| Word Error Rate (WER) | 0.0456 |
| Character Error Rate (CER) | 0.0262 |
| Relative WER Improvement | 86.99% |
| Difficulty | Samples | WER |
|---|---|---|
| Easy | 317 | 0.0303 |
| Medium | 49 | 0.1301 |
| Hard | 7 | 0.2182 |
| Language | Samples | WER |
|---|---|---|
| Hindi | 280 | 0.0514 |
| Hinglish | 93 | 0.0300 |
| Category | Samples | WER |
|---|---|---|
| Perfect | 288 | 0.0384 |
| Minor Phonetic | 50 | 0.0628 |
| Digit Format | 29 | 0.0758 |
| Script Switch | 3 | 0.0682 |
| Genuine Error | 2 | 0.0588 |
| Loanword Mishear | 1 | 0.0000 |
Load the model using the Hugging Face Transformers library.
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
processor = WhisperProcessor.from_pretrained(
"shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)
model = WhisperForConditionalGeneration.from_pretrained(
"shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()
This model is intended for:
| Metric | Value |
|---|---|
| WER | 0.0456 |
| CER | 0.0262 |
| Test Samples | 373 |
| WER Improvement | 86.99% |
This project builds upon:
If you use this model in your research or applications, please cite this Hugging Face repository.
@misc{whisper_medium_hindi_hinglish_asr,
title={Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)},
author={Suja Akhter},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned}}
}
--- language: - hi - en license: apache-2.0 library_name: transformers pipeline_tag: automatic-speech-recognition tags: - automatic-speech-recognition - whisper - speech-recognition - hindi - hinglish - transformers - fine-tuning base_model: openai/whisper-medium --- # Whisper Medium Hindi-Hinglish ASR (Fine-Tuned) A fine-tuned version of **OpenAI Whisper Medium** for **Hindi** and **Hinglish Automatic Speech Recognition (ASR)**. This model was developed to improve Whisper's performance on conversational Hindi and code-mixed Hindi-English speech using a carefully curated synthetic speech corpus. --- # Model Details | Property | Value | |----------|-------| | Base Model | openai/whisper-medium | | Model Type | Whisper Encoder-Decoder Transformer | | Task | Automatic Speech Recognition | | Languages | Hindi, Hinglish | | Framework | Hugging Face Transformers | --- # Motivation OpenAI Whisper provides excellent multilingual speech recognition, but conversational Hindi and Hinglish often remain challenging due to code-mixing, pronunciation variations, and domain-specific vocabulary. This project fine-tunes Whisper Medium using a curated synthetic dataset to significantly improve recognition quality while preserving Whisper's multilingual capabilities. --- # Dataset The training dataset was generated using a multi-stage synthetic data generation pipeline consisting of: - LLM-generated conversational Hindi/Hinglish text - High-quality speech synthesis - Transcript refinement and normalization - Quality filtering - Curated speech-text pairs for Whisper fine-tuning The focus was on maximizing transcript quality while maintaining linguistic diversity. --- # Evaluation Evaluation was performed on an unseen held-out test set. ## Overall Performance | Metric | Score | |---------|-------| | Test Samples | **373** | | Word Error Rate (WER) | **0.0456** | | Character Error Rate (CER) | **0.0262** | | Relative WER Improvement | **86.99%** | --- ## Performance by Difficulty | Difficulty | Samples | WER | |------------|--------:|----:| | Easy | 317 | 0.0303 | | Medium | 49 | 0.1301 | | Hard | 7 | 0.2182 | --- ## Performance by Language | Language | Samples | WER | |----------|--------:|----:| | Hindi | 280 | 0.0514 | | Hinglish | 93 | 0.0300 | --- ## Performance by Error Category | Category | Samples | WER | |----------|--------:|----:| |...
Source context: 64 downloads · 0 likes · Pipeline automatic-speech-recognition · Library transformers · Repo shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned