Loading
Have a ComfyUI workflow? Get a readable Workflow Doc in your browser.Make a workflow readable.Try it now ↗
This repository contains a weights of a scoring model for V(P)AD (Valence, Arousal, and Dominance) emotional state estimation.
Model source
Source description
This repository contains a weights of a scoring model for V(P)AD (Valence, Arousal, and Dominance) emotional state estimation.
The model uses fine-tuned Whisper based encoder to extract audio features and a simple CNN and MLP head to predict VAD scores means and standard deviations.
This repository consists only the CNN-MLP head weights, the encoder weights available at laion/BUD-E-Whisper.
The source code for the model is available at .
Sources
1 sourceVerified Sep 19
Model artifacts
42 artifactsSource excerpts
2 excerptsThe model is intended to be used for estimating VAD scores from audio data, which can be useful in various applications such as emotion recognition, affective computing, and human-computer interaction.
The model was trained with freezed encoder weights on the Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset. The dataset does not have VAD scores provided directly, so to obtain them the following steps were taken:
The rare labels were removed from the training mix and the equal number of samples for each label was used.
v3s_balanced/vad_model_epoch_10_step_1840.pth
pth · 93.8 MB · SHA-256 e1fba8735bfe…1d8c · Hugging Face
Downloadv3s_balanced/vad_model_epoch_11_step_2024.pth
pth · 93.8 MB · SHA-256 d9938ca9f2c9…8c00 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_12_step_2208.pth
pth · 93.8 MB · SHA-256 a96009080302…8db3 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_13_step_2392.pth
pth · 93.8 MB · SHA-256 777ad473c0df…3db9 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_14_step_2576.pth
pth · 93.8 MB · SHA-256 93cdd62af4f9…ee80 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_15_step_2760.pth
pth · 93.8 MB · SHA-256 4824b2e363ae…7bcd · Hugging Face
Downloadv3s_balanced/vad_model_epoch_16_step_2944.pth
pth · 93.8 MB · SHA-256 0ad71bf9770a…20eb · Hugging Face
Downloadv3s_balanced/vad_model_epoch_17_step_3128.pth
pth · 93.8 MB · SHA-256 3009eabf1fa6…2c15 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_18_step_3312.pth
pth · 93.8 MB · SHA-256 c27a88b7af39…73b4 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_19_step_3496.pth
pth · 93.8 MB · SHA-256 7424be77a233…dc22 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_2_step_368.pth
pth · 93.8 MB · SHA-256 0c9c7e1d4873…ed9c · Hugging Face
Downloadv3s_balanced/vad_model_epoch_20_step_3680.pth
pth · 93.8 MB · SHA-256 52a4a4db02b6…89ce · Hugging Face
Downloadv3s_balanced/vad_model_epoch_3_step_552.pth
pth · 93.8 MB · SHA-256 5fdf4557b16a…ac4b · Hugging Face
Downloadv3s_balanced/vad_model_epoch_4_step_736.pth
pth · 93.8 MB · SHA-256 e9d9bd7429e3…48e3 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_5_step_920.pth
pth · 93.8 MB · SHA-256 758dd13b5032…1b1c · Hugging Face
Downloadv3s_balanced/vad_model_epoch_6_step_1104.pth
pth · 93.8 MB · SHA-256 c67aabf3ab07…ce83 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_7_step_1288.pth
pth · 93.8 MB · SHA-256 d5009cc5dea4…2486 · Hugging Face
Downloadv3s_balanced/vad_model_epoch_8_step_1472.pth
pth · 93.8 MB · SHA-256 30c066bfbf0b…a9bf · Hugging Face
Downloadv3s_balanced/vad_model_epoch_9_step_1656.pth
pth · 93.8 MB · SHA-256 7854ebb839fb…ca1f · Hugging Face
Downloadv4_balanced/vad_model_epoch_1_step_2937.pth
pth · 20.0 MB · SHA-256 28316ec7f14c…3c27 · Hugging Face
Downloadv4_balanced/vad_model_epoch_10_step_29370.pth
pth · 20.0 MB · SHA-256 d559ec5b4c13…aa84 · Hugging Face
Downloadv4_balanced/vad_model_epoch_11_step_32307.pth
pth · 20.0 MB · SHA-256 a73d8f0a2535…0012 · Hugging Face
Downloadv4_balanced/vad_model_epoch_12_step_35244.pth
pth · 20.0 MB · SHA-256 516ef50446cc…6527 · Hugging Face
Downloadv4_balanced/vad_model_epoch_13_step_38181.pth
pth · 20.0 MB · SHA-256 3fe5ea56ef81…92c5 · Hugging Face
Downloadv4_balanced/vad_model_epoch_14_step_41118.pth
pth · 20.0 MB · SHA-256 95bd46cc0da7…fd8f · Hugging Face
Downloadv4_balanced/vad_model_epoch_15_step_44055.pth
pth · 20.0 MB · SHA-256 5bb64c293041…de39 · Hugging Face
Downloadv4_balanced/vad_model_epoch_16_step_46992.pth
pth · 20.0 MB · SHA-256 dd0fcf1a2d7a…8cbe · Hugging Face
Downloadv4_balanced/vad_model_epoch_17_step_49929.pth
pth · 20.0 MB · SHA-256 3298f41d3d4d…16fd · Hugging Face
Downloadv4_balanced/vad_model_epoch_18_step_52866.pth
pth · 20.0 MB · SHA-256 beff3a2607df…b095 · Hugging Face
Downloadv4_balanced/vad_model_epoch_19_step_55803.pth
pth · 20.0 MB · SHA-256 8eafc6f13265…1224 · Hugging Face
Downloadv4_balanced/vad_model_epoch_2_step_5874.pth
pth · 20.0 MB · SHA-256 71a75d234a9e…01c6 · Hugging Face
Downloadv4_balanced/vad_model_epoch_20_step_58740.pth
pth · 20.0 MB · SHA-256 bc215bfe2cb5…3240 · Hugging Face
Downloadv4_balanced/vad_model_epoch_3_step_8811.pth
pth · 20.0 MB · SHA-256 5df6a3958ec7…2a69 · Hugging Face
Downloadv4_balanced/vad_model_epoch_4_step_11748.pth
pth · 20.0 MB · SHA-256 6cbc91e8669b…f3e6 · Hugging Face
Downloadv4_balanced/vad_model_epoch_5_step_14685.pth
pth · 20.0 MB · SHA-256 12ee18d06fc9…8923 · Hugging Face
Downloadv4_balanced/vad_model_epoch_6_step_17622.pth
pth · 20.0 MB · SHA-256 2f937dc26248…ef9f · Hugging Face
Downloadv4_balanced/vad_model_epoch_7_step_20559.pth
pth · 20.0 MB · SHA-256 eb65be972c3c…3578 · Hugging Face
Downloadv4_balanced/vad_model_epoch_8_step_23496.pth
pth · 20.0 MB · SHA-256 22696729e5b2…9977 · Hugging Face
Downloadv4_balanced/vad_model_epoch_9_step_26433.pth
pth · 20.0 MB · SHA-256 e7c7834166f5…8673 · Hugging Face
Download--- license: mit pipeline_tag: audio-classification --- # V(P)AD scoring model ## Description This repository contains a weights of a scoring model for V(P)AD (Valence, Arousal, and Dominance) emotional state estimation. The model uses fine-tuned Whisper based encoder to extract audio features and a simple CNN and MLP head to predict VAD scores means and standard deviations. This repository consists only the CNN-MLP head weights, the encoder weights available at [laion/BUD-E-Whisper](https://huggingface.co/laion/BUD-E-Whisper). The source code for the model is available at [this repository](https://github.com/Drakrig/vad_score_prediction). ## Inteneded use The model is intended to be used for estimating VAD scores from audio data, which can be useful in various applications such as emotion recognition, affective computing, and human-computer interaction. ## Data The model was trained with freezed encoder weights on the [Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset](https://huggingface.co/datasets/laion/laions_got_talent_enhanced_flash_annotations_and_long_captions). The dataset does not have VAD scores provided directly, so to obtain them the following steps were taken: 1. Extract emotion categorical labels from the dataset file names. 2. Use Gemma3 27B model to select closest label by meaning from 157 labels described in the [original paper](https://www.researchgate.net/publication/222741832_Evidence_for_a_Three-Factor_Theory_of_Emotions). 3. Use selected labels to obtain VAD scores from the table in the paper. The rare labels were removed from the training mix and the equal number of samples for each label was used. ## Limitations 1. Model was trained and tested on single-speaker audio data, so it may not perform well on multi-speaker audio. 2. Since VAD scores are not directly provided in the dataset, the model's performance may be limited by the quality of the label selection process and the accuracy of the VAD scores obtained from the original paper. 3. Because Whisper usese both semantic and acoustic features, the models performance may be affected by the speaker language if the language is not directly supported by Whisper. ## Credit 1. [LaionAI](https://laion.ai) for the [Laion's Got Talent (Enhanced Flash Annotations and Long Captions) dataset](https://huggingface.co/datasets/laion/laions_got_talent_enhance...
Source context: 0 downloads · 1 likes · Pipeline audio-classification · Repo drakrig/vad_emotion_scorer
