SALMONN-SQA — a calibrated, multi-dimensional speech-quality rater
LoRA/Q-Former weights that turn the SQA-finetuned SALMONN-7B
from a lenient noise specialist into a calibrated, multi-dimensional speech-quality rater.
Given a 16 kHz speech recording it emits per-dimension 1–5 scores, a short grounded description of
what is wrong, and an overall MOS:
noise:2 reverberation:1 bandwidth:4 clipping:5 discontinuity:5 loudness:5.
The recording suffers from heavy reverberation and persistent background noise, and the
limited frequency range gives the voice a somewhat boxed-in, muffled character.
Overall MOS: 1.87
Every input used to train this model is public, so the whole pipeline is reproducible end to end:
LibriTTS-R (CC BY 4.0) + MUSAN (CC BY 4.0) + OpenSLR SLR28 RIRs (Apache-2.0).
The released SQA checkpoint described additive noise well, but as a quality meter it was weak:
a coarse 5-value MOS floored at 2.5, effectively blind to reverberation (ρ(MOS,severity) = −0.11),
and its MOS did not move when a denoiser improved the audio (+0.03).
A Stage-0 linear probe showed the degradation information was present in the frozen encoders
(R² 0.82–0.95 at every tap, including the Q-Former output that reaches the LLM). The blindness lived
in the LLM's learned read-out — trained only on noise — not in the architecture. So the fix was
data + targets, with the encoders left frozen.
Results (held-out)
#
Original SQA
this model
1
MOS↔SNR ρ 0.37
ρ 0.46; names noise 98% at low SNR
2
Calibration vs PESQ/NISQA/DNSMOS ρ 0.40–0.49; 5 distinct MOS values
Reverberation on real measured rooms (held-out RWCP / Aachen AIR / REVERB RIRs, PESQ as an
independent reference): ρ(MOS, PESQ) +0.83 (the older legacy/v3 model scores +0.79).
Files
file
size
what
stage2_checkpoint_best.pth
121 MB
the model you want — scores + description + MOS
stage1_checkpoint_best.pth
121 MB
intermediate calibration-only checkpoint (Stage 2 initialized from it)
legacy/v3_*.pth
121 MB ea.
an earlier variant trained on a non-public RIR set — see the trade-off below
Each contains only the 30.2 M trained parameters — Q-Former (22.8 M), speech→LLaMA projection
(3.1 M), Vicuna LoRA r=8/α=28 (4.2 M), and layer norms. No Vicuna/Whisper/BEATs base weights;
you supply those separately (scripts/setup_salmonn.sh in the repo).
Usage
bash
1git clone https://github.com/LarocheC/llm-sqa &&cd llm-sqa
2uv sync --extra experiments
3bash scripts/setup_salmonn.sh # SALMONN code + Whisper/Vicuna/BEATs4uv run python scripts/fetch_checkpoints.py # pulls this repo5uv run python -m experiments.planb.eval_compare --n-clips 6
How it was trained
Two-stage LoRA-SFT on a 4,000-clip corpus built from LibriTTS-R train-clean-100, degraded on the
fly with known parameters so the targets are exact rather than teacher-labelled.
Targets — per-dimension 1–5 scores derived from the applied parameter (a "severity map");
overall MOS = 0.55·min + 0.45·mean of those scores, blended 70/30 with a fused PESQ+NISQA+DNSMOS
metric MOS; descriptions LLM-paraphrased and programmatically grounding-verified (must name every
degraded axis, and no clean one).
Stage 1 (calibration, 2 epochs) — target is the score block only, so the read-out locks onto
the calibrated numbers with no description tokens competing for the gradient.
Stage 2 (reasoning, 4 epochs) — target is scores + description + MOS, initialized from Stage 1.
Reverberation severity is scored from the worse of the RT60 band and the DRR (direct-to-
reverberant ratio) band. DRR matters as much as RT60: on a RIR set with varying mic distance it
predicts perceived degradation far better (ρ(PESQ,DRR) = +0.67 vs ρ(PESQ,RT60) = −0.27) — a
long-RT60 room still sounds fairly dry when the mic is close. RT60 and DRR are measured from each
impulse response (Schroeder integration), so the pipeline works with any RIR corpus.
Limitations — read before using
The per-dimension structured scores are the most reliable signal. Prefer them over the MOS.
The MOS is calibrated against objective metrics (PESQ/NISQA/DNSMOS), not human listener MOS.
It is not a substitute for a subjective listening test.
Reverberation depends on which reverb you test. Scored against PESQ (an independent reference):
reverb type
legacy/v3
this model
synthetic exp-decay (fixed artificial direct path)
+0.61
+0.39
SLR28 simulated RIRs (image-method) — this model trained on these
+0.33
+0.63
real measured RIRs (RWCP / AIR / REVERB), held out from both
+0.79
+0.83
On genuinely measured rooms the two are close, with this model modestly ahead. The large gap is on
the simulated RIRs it trained on, so that figure reflects in-distribution advantage rather than
generalization. If your audio is dominated by synthetic reverb, evaluate legacy/v3 too.
v3 is also slightly better calibrated on VoiceBank-DEMAND (ρ 0.69–0.75 vs 0.65–0.71) and uses a
finer MOS scale (81 vs 63 distinct values). It is not reproducible from public data.
English, 16 kHz, single-speaker speech. Not validated on music, multi-speaker, or other languages.
License & attribution
Apache-2.0 for these weights. They are a derivative of SALMONN (Apache-2.0) and are used with
Vicuna-7B-v1.5 / Whisper-large-v2 / BEATs, each under its own license — you must obtain those base
models separately and comply with their terms (notably Meta's LLaMA license via Vicuna).
