A Large Audio Language Model (LALM) for Vietnamese speech-to-text, built the cheapest
way possible: bolt a frozen audio encoder onto a small frozen LLM through a thin trainable
projector, then unlock the LLM a little at a time with LoRA.
This repository releases the LoRA + projector adapter (~31 MB) plus a detailed write-up of
the training journey, the problems hit, and the measured results. It is primarily a
documentation / archival release — see Intended use & limitations below.
Tóm tắt (VI): Model ghép audio encoder (FastConformer, đóng băng) + LLM nhỏ (Sailor 0.5B,
đóng băng) qua một projector tuyến tính, rồi mở khóa LLM bằng LoRA. Repo này chia sẻ **adapter
kinh nghiệm + kết quả** của quá trình fine-tune ba nấc. Chi tiết số liệu ở phần Results.
Model summary
Task
Vietnamese automatic speech recognition (with English code-switch and spoken-digit robustness)
The only trainable weights are the projector and the LoRA matrices. The encoder and the LLM
(including its embedding and lm_head) stay frozen — the LLM's 151936-token vocabulary already
covers both Vietnamese and English, which is exactly what a closed-vocabulary CTC/transducer
decoder cannot do.
Intended use & limitations
What this repo is: the trained adapter (adapter.pt) + a full engineering write-up. It is
meant for people studying how to stand up an X-LM (audio/vision/video encoder + LLM) on a
small compute budget, and for archiving the exact weights behind the numbers below.
⚠️ Not runnable stand-alone. The adapter contains only the LoRA deltas and the projector.
To run inference you also need the frozen SALM base: the in-house FastConformer encoder and the
Sailor-0.5B-Chat LLM, wired together with NeMo speechlm2. The frozen encoder checkpoint is
not released here, so this release does not reproduce end-to-end inference on its own.
Domain: tuned for Vietnamese, including conversational/telephony-style speech; English
technical terms are handled via the LLM's open vocabulary.
Not for: safety-critical transcription, languages other than Vietnamese, or as a general
chat model.
Input is audio plus a fixed text prompt containing an audio-locator tag; the projected audio
vectors replace that tag inside the prompt, so the task can be changed by editing the prompt
rather than the architecture. A 10-second utterance becomes ~126 audio frames fed to the LLM.
Training recipe — three-stage curriculum
The guiding principle: unlock parameters in order of return on compute. Align the projector
before letting gradients touch the LLM; open the LLM with ~1% LoRA before considering anything
deeper.
Stage 1 — projector only. Freeze encoder + LLM, align the 0.53M-param projector.
Proves the pipeline runs and gives the next stage a warm projector. Ceiling is low
(WER 27–71%) because a frozen chat LLM prefers to paraphrase rather than transcribe.
Stage 2 — unlock the LLM with LoRA. Insert LoRA (r=16, α=32, dropout 0.05) into all 7
projections per layer (q/k/v/o_proj, gate/up/down_proj). WER drops 2–4× on every set;
English code-switch recall jumps thanks to the LLM's open vocab. Two weaknesses surface:
spoken-digit recall regresses, and the model ignores context-biasing prompts.
Stage 3 — retrain on data that patches the weak spots. Warm-start from the stage-2 adapter;
add synthetic spoken-digit data, widen the broad-domain sets, and interleave prompts carrying a
term list to teach the context-biasing channel. WER improves on all 12 evaluation sets and the
biasing channel starts working.
Optimizer: AdamW, LoRA lr 1e-4, projector lr 2e-4, weight decay 1e-3, cosine schedule with
warmup, batch 16, warm-started between rounds. Because the streaming (lhotse iterable) data has no
declared length, all control is by step, not epoch — stop by max_steps, validate by
integer val_check_interval, and checkpoint the adapter every N steps.
This adapter is the v3 round checkpoint (a full run of the stage-3 curriculum with the
enlarged data mix).
Results
Word Error Rate (%, lower is better) on public Vietnamese benchmarks. LALM and external
baselines are measured on the first 300 utterances per set; the in-house transducer (s7) is
measured on the full suite, so gaps under ~1 point should not be read as real differences.
model
vivos
bud500
cv
fleurs
fosd
lsvsc
vlsp
vietmed
vss
digit_eval
Stage 1 (projector only)
27.56
26.12
44.36
54.07
41.74
52.23
71.25
—
—
—
Stage 2 (LoRA)
9.89
10.36
21.64
18.42
20.41
14.51
30.52
26.10
16.52
22.51
Stage 3 (patched)
7.90
8.01
18.70
15.35
17.59
12.04
26.54
23.89
13.31
12.23
This adapter (v3)
6.78
5.51
17.18
14.58
15.73
10.89
24.79
20.91
11.97
9.00
s7 transducer 114M (ref)
9.23
6.92
18.03
15.81
17.95
12.25
24.83
—
—
—
Qwen3-ASR-0.6B
8.48
6.83
16.55
9.14
13.27
11.35
18.60
22.24
33.07
12.85
parakeet-ctc-0.6b-vi
6.59
7.95
10.73
12.15
8.91
10.24
16.06
23.28
26.72
7.66
chunkformer-large-vie
2.78
4.82
8.39
10.42
5.21
5.98
9.98
15.73
12.42
8.23
Specialized recall (%, higher is better) — English code-switch terms and spoken digits:
model
CS-recall (seen terms)
CS-recall (unseen terms)
digit-recall
This adapter (v3)
52.10
15.38
90.68
— with oracle term hint
—
38.46
—
s7 transducer (closed vocab)
26.74
12.53
—
Qwen3-ASR-0.6B
30.17
31.66
86.74
chunkformer-large-vie (closed vocab)
11.38
6.59
93.99
Reading the numbers:
With ~1.1% trainable parameters, the LALM matches and on 4 sets beats the same-encoder
transducer (s7), and reaches strong WER on the in-house conversational set (vss 11.97).
Open vocabulary is the real differentiator. Code-switch recall (52.10) is nearly double
the closed-vocab transducer (26.74) — chunkformer wins broad-domain WER but its CTC decoder is
stuck at 11.38 on code-switch. This capability cannot be reached by fine-tuning a closed-vocab
model further; it needs the LLM decoder.
Broad-domain Vietnamese still has clear public champions (chunkformer leads 8/12 sets) — the gap
is data and training scale, not architecture.
The context-biasing channel works when trained: feeding the correct term list at inference lifts
unseen-term recall from 15.38 to 38.46 (oracle upper bound).
Key lessons (the "experience" part)
The projector is only a coupling. To change the LLM's behaviour you must let gradients reach
it — a 0.5M-param linear layer alone cannot teach a chat LLM to transcribe verbatim.
Changing the decoder changes the error distribution. Re-measure every specialized skill
(e.g. digit recall) after swapping architecture; the aggregate WER hides regressions.
Measure the oracle before training a control channel. If injecting the answer into the prompt
moves the metric, the channel is worth training; if the oracle does not move, stop early.
Patch weak skills with targeted synthetic data — far cheaper than collecting more real audio.
Synthetic digit data halved WER around digit strings, though lifting exact per-digit recall past
a strong closed-vocab baseline needs more/diverse data.
Control training by step, not epoch, for any length-less streaming data pipeline.
Checkpoint the trained params periodically — the cheapest insurance against infra failures;
here it is a 31 MB file, and it let a hung run resume by warm-starting from the last checkpoint.
Standardize a results registry (one JSON schema) from day one so in-house and external models
live in one comparison table; a new measurement campaign is just adding a row.
The whole recipe is modality-agnostic — swap the audio encoder for a vision/video encoder and
the three-block + curriculum recipe stays the same.
Files in this repo
adapter.pt — the trained adapter (~31 MB): projector weights + LoRA matrices for the frozen
Sailor-0.5B-Chat LLM. Load into a NeMo speechlm2 SALM whose encoder is the frozen FastConformer
and whose LLM is sail/Sailor-0.5B-Chat.
