Du hast einen ComfyUI-Workflow? Erstelle direkt im Browser eine verständliche Workflow-Dokumentation.Workflow verständlich machen.Jetzt ausprobieren ↗
This repository hosts the encoder-decoder VSR model checkpoints introduced in the paper VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness.
Modellquelle
Quellenbeschreibung
This repository hosts the encoder-decoder VSR model checkpoints introduced in the paper VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness.
The models are MultiVSR backbones fine-tuned on the VSRo-200 corpus, a 200-hour collection of Romanian podcast recordings. For training code, data preparation scripts, and inference instructions, please refer to the GitHub repository.
Quellen
1 QuelleVerifiziert 2. Aug.
Modellartefakte
21 Artefaktecheckpoints/model_100h_annot_shuffle2.pt
pt · 1,07 GB · SHA-256 5aedf39154cb…dda3 · Hugging Face
Herunterladencheckpoints/model_100h_annot_shuffle3.pt
pt · 1,07 GB · SHA-256 58fc874d605e…8482 · Hugging Face
HerunterladenQuellenauszüge
2 AuszügeAll checkpoints follow the naming pattern model_[hours]_[type].pt:
_annot — trained on human-annotated transcriptions_auto — trained on automatically generated pseudo-labels_shuffle — alternative data splits used for variance analysis (100h models)_males / _females / _mix — gender-controlled 40h subsets used for bias analysisAll results are reported in Word Error Rate (WER, %) and Character Error Rate (CER, %) on the Test Unseen and Test Seen splits. Lower is better.
| Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) |
|---|---|---|---|---|
| 10h | 72.50 | 41.49 | 67.01 | 37.53 |
| 25h | 64.86 | 36.62 | 59.23 | 32.96 |
| 50h | 58.87 | 33.38 | 54.03 | 29.88 |
| 75h | 54.86 | 30.97 | 51.44 | 28.61 |
| 100h | 53.29 | 29.94 | 48.16 |
| Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) |
|---|---|---|---|---|
| 10h | 74.61 | 42.09 | 68.41 | 38.22 |
| 25h | 66.27 | 37.05 | 60.40 | 33.36 |
| 50h | 59.28 | 33.15 | 55.39 | 30.65 |
| 75h | 56.25 | 31.18 | 51.56 | 28.33 |
| 100h | 53.63 | 30.12 | 49.61 | 27.22 |
A variance analysis across three random shuffles of the 100h subsets yields a mean Word Error Rate (WER) of 53.21% (± 0.37) for the human-annotated data and 53.82% (± 0.17) for the auto-generated data.
| Dataset / Category | # Clips | WER (%) | CER (%) | OOV Token (%) | OOV Type (%) |
|---|---|---|---|---|---|
| Test Seen | 386 | 44.54 | 24.51 | 1.67 | 6.93 |
| Test Unseen | 389 | 48.75 | 27.05 | 2.30 | 8.50 |
| OOD: Vlogs | 99 | 58.61 | 32.85 | 1.49 | 4.26 |
| OOD: Specific domains | 84 | 63.01 | 28.73 | 9.78 | 17.93 |
To evaluate gender bias and cross-speaker generalization, we trained 40-hour baseline models on male-only, female-only, and mixed datasets.
| Training Set (40h) | Global WER (%) | Global CER (%) | Male WER (%) | Male CER (%) | Female WER (%) | Female CER (%) |
|---|---|---|---|---|---|---|
| Males Only | 62.15 | 35.23 | 61.32 | 34.51 | 62.97 | 35.95 |
| Females Only | 59.33 | 33.44 | 59.17 | 32.87 | 59.49 | 34.02 |
| Mixed Data | 59.52 | 33.74 | 59.19 | 33.26 | 59.85 | 34.22 |
| Training Set (40h) | Global WER (%) | Global CER (%) | Male WER (%) | Male CER (%) | Female WER (%) | Female CER (%) |
|---|---|---|---|---|---|---|
| Males Only | 58.82 | 33.11 | 58.58 | 32.59 | 59.06 | 33.63 |
| Females Only | 59.10 | 33.30 | 67.26 | 38.67 | 51.20 | 27.99 |
| Mixed Data | 56.29 | 31.22 | 60.56 | 33.54 | 52.15 | 28.93 |
If you use these models, please cite:
@inproceedings{vsro200,
title = {VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness},
author = {...},
year = {...}
}
checkpoints/model_100h_annot.pt
pt · 1,07 GB · SHA-256 bf4ce940c8dc…f60d · Hugging Face
checkpoints/model_100h_auto_shuffle2.pt
pt · 1,07 GB · SHA-256 8b36d9c9a2aa…66a3 · Hugging Face
Herunterladencheckpoints/model_100h_auto_shuffle3.pt
pt · 1,07 GB · SHA-256 b19fa382ac45…23c1 · Hugging Face
Herunterladencheckpoints/model_40h_females.pt
pt · 1,07 GB · SHA-256 4063d2753f29…d754 · Hugging Face
Herunterladen--- language: - ro library_name: pytorch pipeline_tag: video-text-to-text tags: - visual-speech-recognition - lip-reading - vsr - romanian - speech-recognition - audio-visual datasets: - vsro200/vsro200 metrics: - wer --- # VSRo-200: Romanian Visual Speech Recognition Models This repository hosts the encoder-decoder VSR model checkpoints introduced in the paper *VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness*. The models are MultiVSR backbones fine-tuned on the **VSRo-200** corpus, a 200-hour collection of Romanian podcast recordings. For training code, data preparation scripts, and inference instructions, please refer to the [GitHub repository](https://github.com/vsro200/vsro200). ## Checkpoints All checkpoints follow the naming pattern `model_[hours]_[type].pt`: - `_annot` — trained on human-annotated transcriptions - `_auto` — trained on automatically generated pseudo-labels - `_shuffle` — alternative data splits used for variance analysis (100h models) - `_males` / `_females` / `_mix` — gender-controlled 40h subsets used for bias analysis ## Results All results are reported in Word Error Rate (WER, %) and Character Error Rate (CER, %) on the **Test Unseen** and **Test Seen** splits. Lower is better. #### Human Annotated Data | Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) | |:---:|:---:|:---:|:---:|:---:| | 10h | 72.50 | 41.49 | 67.01 | 37.53 | | 25h | 64.86 | 36.62 | 59.23 | 32.96 | | 50h | 58.87 | 33.38 | 54.03 | 29.88 | | 75h | 54.86 | 30.97 | 51.44 | 28.61 | | 100h | **53.29** | **29.94** | **48.16** | **26.53** | #### Whisper Pseudo Labels | Training Hours | Test Unseen WER (%) | Test Unseen CER (%) | Test Seen WER (%) | Test Seen CER (%) | |:---:|:---:|:---:|:---:|:---:| | 10h | 74.61 | 42.09 | 68.41 | 38.22 | | 25h | 66.27 | 37.05 | 60.40 | 33.36 | | 50h | 59.28 | 33.15 | 55.39 | 30.65 | | 75h | 56.25 | 31.18 | 51.56 | 28.33 | | 100h | 53.63 | 30.12 | 49.61 | 27.22 | | 125h | 51.71 | 29.04 | 48.68 | 26.58 | | 150h | 51.25 | 28.40 | 47.05 | 25.64 | | 175h | 49.84 | 27.66 | 46.44 | 25.30 | | 200h | **48.75** | **27.05** | **44.54** | **24.51** | A variance analysis across three random shuffles of the 100h subsets yields a mean Word Error Rate (WER) of 53.21% (± 0.37) for the human-annotated data and 53.82% (± 0.17) for the...
Source context: 0 downloads · 0 likes · Pipeline video-text-to-text · Library pytorch · Repo vsro200/models-vsro200
| 125h | 51.71 | 29.04 | 48.68 | 26.58 |
| 150h | 51.25 | 28.40 | 47.05 | 25.64 |
| 175h | 49.84 | 27.66 | 46.44 | 25.30 |
| 200h | 48.75 | 27.05 | 44.54 | 24.51 |
| OOD: Noisy |
| 100 |
| 68.96 |
| 33.68 |
| 6.19 |
| 12.88 |
| OOD: Archival | 92 | 87.97 | 50.44 | 5.24 | 10.96 |
| Global OOD | 375 | 68.46 | 35.99 | 5.08 | 14.75 |