Audio Audio Majestrino With Emotion And Quality Experts

Audio-Audio Majestrino with Emotion and Quality Experts

A ViT-Base audio encoder trained via contrastive learning on 100 million DACVAE-encoded audio samples, paired with lightweight MLP expert heads for emotion recognition, speaker embedding prediction, and audio quality assessment.

Overview

This model operates in the DACVAE latent space rather than on raw waveforms. Audio is first encoded to DACVAE latents (25fps × 128-dim float vectors), then the ViT-Base encoder produces a 512-dimensional embedding. Specialized MLP heads predict various audio attributes from this embedding.

Architecture

Raw Audio (48kHz)
    ↓
DACVAE Encoder (facebook/dacvae-watermarked)
    ↓
DACVAE Latents (T × 128, float16, 25fps)
    ↓
ViT-Base Encoder (86M params)
  - 1D patch embedding: Conv1d(128→768, kernel=5, stride=5)
  - 12 transformer layers (768-dim, 12 heads, 3072 MLP dim)
  - CLS token pooling → LayerNorm → Linear(768→512) → L2-normalize
    ↓
512-dim Audio Embedding
    ↓
Expert MLP Heads:
  ├── Emotion Expert (144K params)    → 40 emotion predictions (best)
  ├── Attribute Expert (60K params)   → 13 non-emotion attribute predictions (best)
  ├── All-53 Expert (212K params)     → 53 combined predictions (alternative)
  ├── Attribute Probe (898K params)   → 53 emotion/speaker attributes + duration
  ├── Speaker Probe (950K params)     → 128-dim timbre embedding
  ├── 5× Quality Experts (37K each)   → individual quality scores
  └── Distortion Expert (74K params)  → binary clean vs distorted (BCE)

Models Included

Model	File	Params	Description
ViT-Base Encoder	encoder.pt	86M	Core audio encoder, produces 512-dim embeddings
Emotion Expert (40)	emotion_expert_emo40.pt	144K	Best for 40 emotion dimensions (MAE 0.345)
Attribute Expert (13)	emotion_expert_attr13.pt	60K	Best for 13 non-emotion attributes (MAE 0.507)
All-53 Expert	emotion_expert_all53.pt	212K	All 53 attributes in one model (MAE 0.387)
Attribute Probe	attribute_probe.pt	898K	Predicts 53 emotion/speaker attributes + duration
Speaker Probe	speaker_probe.pt	950K	Predicts 128-dim wavelm timbre embedding
Quality: CPS	quality_expert_cps.pt	37K	Characters per second (speech rate)
Quality: Background	quality_expert_score_background_quality.pt	37K	Background noise quality
Quality: Content	quality_expert_score_content_enjoyment.pt	37K	Content enjoyment
Quality: Overall	quality_expert_score_overall_quality.pt	37K	Overall audio quality
Quality: Speech	quality_expert_score_speech_quality.pt	37K	Speech quality
Distortion Expert	distortion_expert.pt	74K	Binary clean vs. distorted classifier (92.5% acc, 0.984 ROC-AUC)

Training Details

ViT-Base Encoder (Contrastive Pre-training)

The encoder was trained using symmetric InfoNCE contrastive loss with data augmentation on DACVAE latents:

• Training data: 100M samples from TTS-AGI/maestrino-data-DACVAE, TTS-AGI/enhanced-audiosnippets-DACVAE, and TTS-AGI/emotion-attribute-conditioning-dacvae
• Effective batch size: 8,192 (128 per GPU × 8 GPUs × 8 gradient accumulation steps)
• Optimizer: AdamW (lr=5e-4, betas=(0.9, 0.98), weight_decay=0.05)
• Schedule: Cosine annealing with 5% warmup
• Augmentations: Random split (40%), frame drop (30%), random crop (30%)
• Training steps: 12,207 (100M samples seen)
• Hardware: 8× GPU distributed training with bfloat16 mixed precision

Final validation metrics (audio-to-audio retrieval, 3K samples):

Metric	Value
Drop→Complete Top-1	99.1%
Drop→Complete Top-5	100%
Complete→Drop Top-1	99.2%
Complete→Drop Top-5	100%
Half1→Half2 Top-1	91.7%
Half1→Half2 Top-3	97.8%

Emotion Experts v2 (Multi-Output MLPs)

Three multi-output MLP experts trained on 220K merged samples (68K emotion-attribute-conditioning + 159K balanced-audio-snippets, deduplicated). Evaluated on a common held-out 2000-sample validation set.

53 Attributes (scale 0-4):

• 40 emotions: Affection, Amusement, Anger, Astonishment/Surprise, Awe, Bitterness, Concentration, Confusion, Contemplation, Contempt, Contentment, Disappointment, Disgust, Distress, Doubt, Elation, Embarrassment, Emotional Numbness, Fatigue/Exhaustion, Fear, Helplessness, Hope/Enthusiasm/Optimism, Impatience and Irritability, Infatuation, Interest, Intoxication/Altered States, Jealousy & Envy, Longing, Malevolence/Malice, Pain, Pleasure/Ecstasy, Pride, Relief, Sadness, Sexual Lust, Shame, Sourness, Teasing, Thankfulness/Gratitude, Triumph
• 10 speaker/voice attributes: Age, Gender, Confident vs. Hesitant, High-Pitched vs. Low-Pitched, Monotone vs. Expressive, Serious vs. Humorous, Soft vs. Harsh, Submissive vs. Dominant, Vulnerable vs. Emotionally Detached, Warm vs. Cold
• 3 dimensional: Valence, Arousal, Authenticity

Model	Architecture	Params	Outputs	Mean MAE
Emotion-40 (best for emotions)	512→192→192→40	144K	40 emotions	0.345
Attribute-13 (best for non-emotions)	512→96→96→13	60K	13 non-emotion attrs	0.507
All-53 (combined)	512→256→256→53	212K	All 53 attributes	0.387

• Training: 200 epochs, Huber loss (delta=1.0), AdamW (lr=1e-3, weight_decay=1e-4), CosineAnnealingLR, batch_size=4096
• Architecture: Linear→LayerNorm→GELU→Dropout(0.1) × 2 hidden layers → Linear→output_dim
• The Emotion-40 model wins 31/53 attributes, All-53 wins 19/53, Attribute-13 wins 3/53

For best per-attribute performance, use Emotion-40 for the 40 emotion dims and Attribute-13 for the 13 non-emotion dims.

Per-attribute MAE (best model per attribute, sorted best to worst):

Attribute	Best MAE	Model	Attribute	Best MAE	Model
Jealousy & Envy	0.099	emo40	Contemplation	0.429	emo40
Embarrassment	0.151	emo40	Affection	0.435	all53
Infatuation	0.156	emo40	Amusement	0.284	emo40
Shame	0.155	emo40	Elation	0.443	emo40
Intoxication	0.165	emo40	Pleasure/Ecstasy	0.447	all53
Sexual Lust	0.184	emo40	Disappointment	0.479	emo40
Sourness	0.207	emo40	Impatience	0.522	emo40
Disgust	0.222	emo40	Arousal	0.522	all53
Fatigue/Exhaustion	0.224	emo40	Confident vs. Hesitant	0.521	all53
Relief	0.237	emo40	Interest	0.537	emo40
Fear	0.257	emo40	Concentration	0.553	all53
Teasing	0.255	emo40	Hope/Enthusiasm	0.558	emo40
Pitch (High vs. Low)	0.273	all53	Sadness	0.564	emo40
Pain	0.274	emo40	Distress	0.559	emo40
Astonishment/Surprise	0.299	emo40	Gender	0.597	all53
Doubt	0.302	emo40	Warm vs. Cold	0.571	all53
Malevolence/Malice	0.301	all53	Submissive vs. Dominant	0.468	all53
Bitterness	0.312	emo40	Vulnerable vs. Detached	0.541	all53
Confusion	0.314	emo40	Monotone vs. Expressive	0.406	attr13
Authenticity	0.331	all53	Age	0.451	attr13
Contempt	0.352	all53	Valence	0.985	attr13
Triumph	0.373	emo40	Soft vs. Harsh	0.448	all53
Helplessness	0.404	emo40	Serious vs. Humorous	0.429	all53
Contentment	0.431	all53	Longing	0.408	emo40
Emotional Numbness	0.369	emo40	Pride	0.452	all53
Anger	0.433	emo40	Thankfulness/Gratitude	0.443	all53

Attribute Probe (Two-Phase Training)

Predicts 53 attributes from the 512-dim embedding using a shared backbone with separate heads for attributes and duration. This is a legacy model — the v2 Emotion Experts above achieve better per-attribute MAE.

Phase 1 (pre-training): 10 epochs on TTS-AGI/emotion-attribute-conditioning-dacvae (68K samples) Phase 2 (fine-tuning): 50 epochs on TTS-AGI/balanced-audio-snippets-40x3k-DACVAE (132K) + TTS-AGI/emolia-3k-speaker-clusters-DACVAE (27K)

• Architecture: Linear(512→704)→LN→GELU→Dropout(0.1)→Linear(704→704)→LN→GELU→Dropout(0.1), attr_head(704→53), dur_head(704→1)
• Loss: Huber (delta=1.0)
• Optimizer: AdamW (lr=1e-3, weight_decay=1e-4), CosineAnnealingLR
• Best val loss: 0.459 (Phase 2, epoch 41)
• Mean attribute MAE: 0.398

Speaker Probe (Two-Phase Training)

Predicts 128-dim wavelm timbre embedding from the 512-dim audio embedding.

Phase 1: 10 epochs on emotion-attribute-conditioning-dacvae (68K samples) Phase 2: 30 epochs on TTS-AGI/emolia-3k-speaker-clusters-DACVAE (63K samples, 3000 speaker clusters)

• Architecture: Linear(512→704)→LN→GELU→Dropout(0.1)→Linear(704→704)→LN→GELU→Dropout(0.1)→Linear(704→128)
• Val split: 1 sample per cluster (3000 val samples)
• Best val loss: 0.00236
• Cosine similarity: 0.625
• MAE: 0.054

Quality Experts

Five independent small MLPs trained on TTS-AGI/balanced-audio-score-datasets-DACVAE.

Expert	Val MAE	Val Loss	Description
CPS	4.638	4.198	Characters per second (speech rate)
Background Quality	0.292	0.087	Background noise quality (0-4 scale)
Content Enjoyment	0.483	0.188	Content enjoyment rating (0-8.6 scale)
Overall Quality	0.258	0.058	Overall audio quality (0-3.7 scale)
Speech Quality	0.277	0.078	Speech quality (0-3.9 scale)

• Architecture: Linear(512→64)→LN→GELU→Dropout(0.1)→Linear(64→64)→LN→GELU→Dropout(0.1)→Linear(64→1)
• Training: 50 epochs, Huber loss, AdamW (lr=1e-3), CosineAnnealingLR, batch_size=4096

Distortion Expert (Binary Clean vs. Distorted Classifier)

A lightweight binary classifier trained on top of the frozen majestrino encoder to discriminate clean speech from artificially degraded speech. The primary use case is as a fast quality filter for generated TTS output and for cleaning training corpora — it detects signal-level artifacts (clipping, comb-filtering) that the existing DNSMOS-based quality experts may not catch cleanly.

The expert outputs a single logit; apply sigmoid to get P(clean). For absolute quality filtering of natural speech, threshold at 0.5. For relative ranking of TTS-generated samples (which are out-of-distribution for the training set), rank by the raw logit — higher is better.

Training data: 100K clean/distorted pairs (200K total samples) built from laion/emolia-hq English standard-HQ tars. Each clean clip (8 seconds, 48 kHz mono) is paired with exactly one distorted twin. Three distortion families are applied in a deterministic...