--- library_name: pytorch tags: - audio-processing - music-classification - math-rock - midwest-emo - pytorch - wavlm - clap - transformer datasets: - anggars/neural-mathrock language: - en - id - ja metrics: - accuracy - loss - recall - f1 pipeline_tag: audio-classification --- # Neural Math Rock: Dual-Backbone Acoustic and Affective Analysis Pipeline This repository hosts an advanced deep learning framework specialized in the acoustic, structural, and affective analysis of Math Rock and Midwest Emo music. By integrating raw waveform transformers and contrastive audio-language pre-trained encoders, the system extracts high-fidelity representations to decode intricate musical arrangements and complex emotional states. ## Project Objectives The framework is engineered to handle the signature complexities of math rock—such as highly technical guitar tapping, syncopated drum patterns, non-standard time signatures, and sudden dynamic shifts: - **Multi-Task Classification:** Simultaneous prediction of Emotion (28 multi-label classes), Aesthetic Vibe (4 classes), Dynamic Intensity (3 classes), and Tempo (3 classes). - **Acoustics-to-Affect Mapping:** Native modeling of raw audio signals to capture the subtle interactions between complex musical textures and implicit vocal deliveries. - **Multimodal Integration Ready:** Architecture optimized for late-fusion coupling with cross-lingual semantic text models (e.g., XLM-RoBERTa) for hybrid audio-lyrical downstream analysis. ## Technical Architecture The model discards legacy 2D-CNN log-mel spectrogram approaches in favor of a fully unfrozen dual-transformer backbone that processes raw audio waveforms natively through an integrated downstream projection and late-fusion network. ``` +------------------+ | Raw Audio Wave | +--------+---------+ | +---------------------+---------------------+ | | v v +-----------------+ +-----------------+ | WavLM-Base | | CLAP Audio | | (Speech/Vocal) | | (Acoustic/Sfx) | +--------+--------+ +--------+--------+ | | v v +-----------------+ +-----------------+ | Linear Project | | Linear Project | | (768 -> 512) | | (768 -> 512) | +--------+--------+ +--------+--------+ | | +---------------------+---------------------+ | v +-----------------+ | Feature Concat | | (1024-dim) | +--------+--------+ | v +-----------------+ | LayerNorm + Tanh| +--------+--------+ | +---------------------+---------------------+---------------------+ | | | | v v v v +-----------------+ +-----------------+ +-----------------+ +-----------------+ | Emotion Head | | Vibe Head | | Intensity Head | | Tempo Head | | (28 Classes) | | (4 Classes) | | (3 Classes) | | (3 Classes) | +-----------------+ +-----------------+ +-----------------+ +-----------------+ ``` ### 1. Acoustic Stream (Dual-Backbone Encoders) - **Speech & Vocal Intonation Backbone:** `microsoft/wavlm-base` processes raw waveforms to capture structural temporal context, pitch contours, and vocal delivery dynamics. - **General Audio & Timbre Backbone:** `laion/clap-htsat-unfused` processes raw waveforms via an unfused HTSAT topology to capture global acoustic textures, instrumental signatures, and overall arrangement timbre. - **Downstream Embeddings Layer:** Features from both backbones are mapped via independent linear layers (`wlm_proj`, `clp_proj`) to a stabilized 512-dimensional subspace before concatenation. ### 2. Multi-Task Fusion & Head Layers - **Fusion Network:** Concat embeddings (1024-dim) are bound through a regularized `LayerNorm` $\rightarrow$ `Tanh` $\rightarrow$ `Dropout(0.3)` projection pipeline to prevent intermediate feature exploding. - **Classification Heads:** Fully connected downstream networks mapping the unified representation to separate multi-class and independent multi-label targets. ## Taxonomy Specifications ### Affective & Structural Targets - **Emotion (28 Classes, Independent Multi-Label):** `admiration`, `amusement`, `anger`, `annoyance`, `approval`, `caring`, `confusion`, `curiosity`, `desire`, `disappointment`, `disapproval`, `disgust`, `embarrassment`, `excitement`, `fear`, `gratitude`, `grief`, `joy`, `love`, `nervousness`, `optimism`, `pride`, `realization`, `relief`, `remorse`, `sadness`, `surprise`, `neutral`. - **Vibe (4 Classes):** `aggressive`, `atmospheric`, `melancholic`, `technical`. - **Intensity Level (3 Classes):** `low`, `medium`, `high`. - **Tempo Classification (3 Classes):** `slow`, `moderate`, `fast`. ## Hyperparameters & Training Configuration The framework was trained across 20 epochs using an accelerated multi-task dynamic gradient pipeline: - **Optimization:** `AdamW` with discriminative learning rates ($3\times10^{-5}$ for base transformers, $1\times10^{-4}$ for projection blocks and multi-task heads) and a weight decay coefficient of $0.01$. - **Loss Framework:** Independent Binary Cross-Entropy (BCE) Focal Loss ($\gamma = 2.0$, label smoothing = $0.1$) applied to the multi-label emotion block to stabilize highly imbalanced heads, coupled with weighted Cross-Entropy for categorical tracks. - **Regularization:** Gradient norm clipping capped at a maximum threshold of $1.0$; mixed-precision training enabled via `torch.amp`. ## How to Use Because this model uses a custom dual-transformer architecture, the model class must be declared locally prior to initializing the state dictionary checkpoints. ```python import torch import torch.nn as nn import torch.nn.functional as F from transformers import WavLMModel, ClapAudioModel class AudioMathRockModel(nn.Module): def __init__(self) -> None: super().__init__() # Initialize pretrained dual backbone transformers self.wavlm = WavLMModel.from_pretrained("microsoft/wavlm-base") self.clap = ClapAudioModel.from_pretrained("laion/clap-htsat-unfused") # Downstream embedding projection layers self.wlm_proj = nn.Linear(768, 512) self.clp_proj = nn.Linear(768, 512) # Non-linear fusion pipeline self.fusion = nn.Sequential( nn.Linear(1024, 512), nn.LayerNorm(512), nn.Tanh(), nn.Dropout(0.3), ) # Multi-task classification networks self.emo_head = nn.Sequential( nn.Linear(512, 256), nn.GELU(), nn.Dropout(0.2), nn.Linear(256, 28), # 28 independent multi-label emotions ) self.vibe_head = nn.Linear(512, 4) # 4 acoustic vibes self.int_head = nn.Linear(512, 3) # 3 intensity classes self.tmp_head = nn.Linear(512, 3) # 3 tempo classes def forward(self, wavlm_values: torch.Tensor, clap_values: torch.Tensor) -> tuple: # Extract temporal mean features from WavLM and pooled features from CLAP wlm_feats = self.wavlm(wavlm_values).last_hidden_state.mean(dim=1) clp_feats = self.clap(clap_values).pooler_output # Project features into balanced dimension space wlm_p = F.gelu(self.wlm_proj(wlm_feats)) clp_p = F.gelu(self.clp_proj(clp_feats)) # Perform feature concatenation and multi-head prediction fused = self.fusion(torch.cat([wlm_p, clp_p], dim=-1)) return self.emo_head(fused), self.vibe_head(fused), self.int_head(fused), self.tmp_head(fused) # Initialize and load model checkpoints device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = AudioMathRockModel().to(device) checkpoint = torch.load("model.pth", map_location=device) model.load_state_dict(checkpoint["model_state_dict"]) model.eval() ```
