Spee Project Fairhindiser Clues

--- language: - en - hi tags: - audio - speech - emotion-recognition - fairness - robustness - explainability license: mit datasets: - iemocap - iitkgp base_model: facebook/wav2vec2-base library_name: pytorch pipeline_tag: audio-classification --- # FairHindiSER: Fair and Robust Speech Emotion Recognition This repository contains the checkpoints for **FairHindiSER**, a speech emotion recognition model trained on a 50/50 mix of Hindi (IITKGP) and English (IEMOCAP) emotional speech. The model predicts four emotions: *angry, happy, neutral, sad*. The backbone is [`facebook/wav2vec2-base`](https://huggingface.co/facebook/wav2vec2-base), adapted with: - A FairSER MLP head on top of pooled wav2vec2 features. - LoRA adapters on the top transformer layers. - CLUES-style contrastive debiasing to reduce gaps across language/gender groups. - Gradual full unfreezing of the encoder with Optuna-tuned learning rates. ## Checkpoints in this repo - `checkpoints/head_best.pt` — head-only fine-tuning (backbone frozen). - `checkpoints/lora_best.pt` — LoRA fine-tuning with focal loss and class weights. - `checkpoints/clues_lora_best.pt` — LoRA + CLUES contrastive debiasing. - `checkpoints/full_best.pt` — final model with gradual full unfreezing. You can load the final model like this: ```python import torch from transformers import Wav2Vec2Model from fairhindiser import FairSERModel # your model class, if you publish it as a pip package ckpt = torch.load("checkpoints/full_best.pt", map_location="cpu") model = FairSERModel() model.load_state_dict(ckpt) model.eval() ``` (Adapt the import path to your own project structure.) ## Intended use - Research on cross-lingual and fair speech emotion recognition. - Analysis of robustness and calibration under common audio corruptions (noise, speed, pitch perturbations). - As a backbone for downstream SER systems in multilingual / accented settings. Not intended for high-stakes decisions or medical/psychological diagnosis. ## Training data - **Hindi:** IITKGP Hindi corpus (naturalistic, acted dialogues). - **English:** IEMOCAP 4-class subset (angry, happy/excited, neutral, sad). All audio was resampled to 16 kHz mono and normalized. The combined dataset contains 3200 Hindi + 3200 English clips, stratified into train/val/test splits. ## Evaluation We report metrics on the held-out test set and AudioTrust-style axes: - *...