Salmonn Sqa Planb v3

SALMONN-SQA — a calibrated, multi-dimensional speech-quality rater

LoRA/Q-Former weights that turn the SQA-finetuned SALMONN-7B from a lenient noise specialist into a calibrated, multi-dimensional speech-quality rater.

Given a 16 kHz speech recording it emits per-dimension 1–5 scores, a short grounded description of what is wrong, and an overall MOS:

noise:2 reverberation:1 bandwidth:4 clipping:5 discontinuity:5 loudness:5.
The recording suffers from heavy reverberation and persistent background noise, and the
limited frequency range gives the voice a somewhat boxed-in, muffled character.
Overall MOS: 1.87

Every input used to train this model is public, so the whole pipeline is reproducible end to end: LibriTTS-R (CC BY 4.0) + MUSAN (CC BY 4.0) + OpenSLR SLR28 RIRs (Apache-2.0).

Code, corpus and the full experiment log: https://github.com/LarocheC/llm-sqa

What problem this fixes

The released SQA checkpoint described additive noise well, but as a quality meter it was weak: a coarse 5-value MOS floored at 2.5, effectively blind to reverberation (ρ(MOS,severity) = −0.11), and its MOS did not move when a denoiser improved the audio (+0.03).

A Stage-0 linear probe showed the degradation information was present in the frozen encoders (R2 0.82–0.95 at every tap, including the Q-Former output that reaches the LLM). The blindness lived in the LLM's learned read-out — trained only on noise — not in the architecture. So the fix was data + targets, with the encoders left frozen.

Results (held-out)

#	Original SQA	this model
1	MOS↔SNR ρ 0.37	ρ 0.46; names noise 98% at low SNR
2	Calibration vs PESQ/NISQA/DNSMOS ρ 0.40–0.49; 5 distinct MOS values	ρ 0.65–0.71; 63 distinct values (1.90–4.89)
3	reverb −0.27, bandwidth −0.37, clipping −0.48	reverb −0.80, bandwidth −0.88, clipping −0.81, noise −0.93
4	Enhancement: MOS +0.03, ρ(gain,PESQ) ≈ 0	MOS gain +1.05 (91% of files), ρ +0.32
5	Degenerate/made-up JSON under OOD input	0/108 degenerate

Reverberation on real measured rooms (held-out RWCP / Aachen AIR / REVERB RIRs, PESQ as an independent reference): ρ(MOS, PESQ) +0.83 (the older legacy/v3 model scores +0.79).

Files

file	size	what
stage2_checkpoint_best.pth	121 MB	the model you want — scores + description + MOS
stage1_checkpoint_best.pth	121 MB	intermediate calibration-only checkpoint (Stage 2 initialized from it)
legacy/v3_*.pth	121 MB ea.	an earlier variant trained on a non-public RIR set — see the trade-off below

Each contains only the 30.2 M trained parameters — Q-Former (22.8 M), speech→LLaMA projection (3.1 M), Vicuna LoRA r=8/α=28 (4.2 M), and layer norms. No Vicuna/Whisper/BEATs base weights; you supply those separately (scripts/setup_salmonn.sh in the repo).

Usage

git clone https://github.com/LarocheC/llm-sqa && cd llm-sqa
uv sync --extra experiments
bash scripts/setup_salmonn.sh                       # SALMONN code + Whisper/Vicuna/BEATs
uv run python scripts/fetch_checkpoints.py          # pulls this repo
uv run python -m experiments.planb.eval_compare --n-clips 6

How it was trained

Two-stage LoRA-SFT on a 4,000-clip corpus built from LibriTTS-R train-clean-100, degraded on the fly with known parameters so the targets are exact rather than teacher-labelled.

• Degradations — reverb (SLR28 simulated RIRs + our synthetic exp-decay), noise (real MUSAN + synthetic white/pink/brown), codec (Opus/MP3), band-limiting, clipping, packet loss, loudness. A blind-spot-weighted sampler gives each clip 0–3 axes; ~46% of clips carry ≥2 simultaneously.
• Targets — per-dimension 1–5 scores derived from the applied parameter (a "severity map"); overall MOS = 0.55·min + 0.45·mean of those scores, blended 70/30 with a fused PESQ+NISQA+DNSMOS metric MOS; descriptions LLM-paraphrased and programmatically grounding-verified (must name every degraded axis, and no clean one).
• Stage 1 (calibration, 2 epochs) — target is the score block only, so the read-out locks onto the calibrated numbers with no description tokens competing for the gradient.
• Stage 2 (reasoning, 4 epochs) — target is scores + description + MOS, initialized from Stage 1.

Trainable: Q-Former + projection + LoRA. Frozen: Whisper, BEATs, Vicuna base.

Reverberation severity is scored from the worse of the RT60 band and the DRR (direct-to- reverberant ratio) band. DRR matters as much as RT60: on a RIR set with varying mic distance it predicts perceived degradation far better (ρ(PESQ,DRR) = +0.67 vs ρ(PESQ,RT60) = −0.27) — a long-RT60 room still sounds fairly dry when the mic is close. RT60 and DRR are measured from each impulse response (Schroeder integration), so the pipeline works with any RIR corpus.

Limitations — read before using

• The per-dimension structured scores are the most reliable signal. Prefer them over the MOS.

• The MOS is calibrated against objective metrics (PESQ/NISQA/DNSMOS), not human listener MOS. It is not a substitute for a subjective listening test.

• Reverberation depends on which reverb you test. Scored against PESQ (an independent reference):

  reverb type	legacy/v3	this model
  synthetic exp-decay (fixed artificial direct path)	+0.61	+0.39
  SLR28 simulated RIRs (image-method) — this model trained on these	+0.33	+0.63
  real measured RIRs (RWCP / AIR / REVERB), held out from both	+0.79	+0.83

  On genuinely measured rooms the two are close, with this model modestly ahead. The large gap is on the simulated RIRs it trained on, so that figure reflects in-distribution advantage rather than generalization. If your audio is dominated by synthetic reverb, evaluate legacy/v3 too.

• v3 is also slightly better calibrated on VoiceBank-DEMAND (ρ 0.69–0.75 vs 0.65–0.71) and uses a finer MOS scale (81 vs 63 distinct values). It is not reproducible from public data.

• English, 16 kHz, single-speaker speech. Not validated on music, multi-speaker, or other languages.

License & attribution

Apache-2.0 for these weights. They are a derivative of SALMONN (Apache-2.0) and are used with Vicuna-7B-v1.5 / Whisper-large-v2 / BEATs, each under its own license — you must obtain those base models separately and comply with their terms (notably Meta's LLaMA license via Vicuna).