A LoRA adapter for Qwen/Qwen3-ASR-1.7B-hf, fine-tuned on a 34-hour Vietnamese mixture of four sources — viVoice (YouTube), VIVOS (read speech), Common Voice 17 vi (crowd-sourced) and VLSP 2020 (VinBigData 100h).

This is not a full model: it is 69 MB of adapter weights (17.4 M trainable parameters, 0.85% of the base) that must be loaded on top of the base checkpoint. The audio encoder is frozen — only the 28 decoder layers are adapted, on all four attention projections plus all three MLP projections (196 modules).

The adapter's job is domain adaptation, not general capability: the base model is already strong on Vietnamese, and what 34 hours buys is a better fit to the acoustic and transcript conventions of Vietnamese corpora.

⚠️ Punctuation is domain-conditional — read Output style before adopting. On read-speech and spontaneous audio this adapter emits unpunctuated text where the base model punctuates. It does not affect WER/CER, but it will affect you.

Trained on 5–60 s segments, so it is at its best on 5–30 s audio. Very short clips (< 5 s) are the one bucket where it regresses against the base model — see 🎯 Accuracy below.

🚋 Usage

Below is a quick way to get up and running with the model.

Install

pip install "transformers>=5.14" peft torch torchaudio soundfile

transformers>=5.14 is required — the qwen3_asr architecture was added after 5.5.x, and earlier versions raise KeyError: 'qwen3_asr'.

Load the base model and attach the adapter

python

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor
from peft import PeftModel

BASE = "Qwen/Qwen3-ASR-1.7B-hf"
ADAPTER = "tyanfarm/Qwen3-ASR-1.7B-34h"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForMultimodalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, attn_implementation="sdpa", device_map="cuda",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# Batched generation is only correct with left padding.
processor.tokenizer.padding_side = "left"

Transcribe

python

import librosa

# Pass a decoded 16 kHz float array, not a file path: the processor's own file
# loader goes through torchcodec+FFmpeg, and FFmpeg 4 hits a "0 channels" bug.
audio, _ = librosa.load("sample.wav", sr=16000)

inputs = processor.apply_transcription_request(
    audio=[audio], language="Vietnamese",
).to(model.device, model.dtype)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=440, do_sample=False)

gen = out[:, inputs["input_ids"].shape[1]:]
print(processor.decode(gen, return_format="transcription_only")[0])
# 'à những cái cảm thẳn trong dòng huế thiên liệu như các bạn đã từng đến huế...'
# Note the bare, unpunctuated style on this kind of audio -- see "Output style" below.

Batching. Sort by length and batch — with padding_side="left" set above, audio=[a1, a2, ...] transcribes several clips per call. Mixing a 60 s clip with a 2 s one wastes most of the batch on padding, so bucket by duration.

Long audio. Chunk it. The adapter saw nothing longer than 60 s in training, and 25 s VAD-aligned chunks are what this model was validated on.

📡 Training data

A 34-hour, four-source mixture (19,881 clips after filtering the 4 clips longer than 60 s), resampled to 16 kHz:

source	slice used	clips	hours
viVoice	YouTube, 5–60 s merged segments, channel-disjoint train split	980	6.41
VIVOS	official `train`	11,660	14.92
Common Voice 17 `vi`	official `train`	2,298	2.90
VLSP 2020 (VinBigData 100h)	90% of an 11 h stream, assigned by `md5(transcript)`	4,943	9.78
total		19,881	34.00

Transcript casing was normalized on ingest. VIVOS ships ALL CAPS and is 59% of the mixture; an earlier run left that alone and the model learned to condition casing on acoustic domain — all 760 VIVOS test clips came back uppercase and VIVOS WER went 7.19% → 11.64%. That is a tokenizer effect: uppercase Vietnamese has no whole-word tokens in the Qwen vocabulary ("VÀ NẾU BẠN" is 8 tokens vs 3 for "và nếu bạn"), so generation runs near character granularity. All-caps rows are folded to ordinary case; real proper-noun capitals survive. Punctuation is left as each source ships it — which is where the domain-conditional punctuation described under 🖋 Output style comes from.

Hyperparameters


LoRA rank / alpha / dropout	16 / 32 / 0.05
target modules	`language_model.layers.*.{q,k,v,o}_proj` + `mlp.{gate,up,down}_proj` (196)
trainable params	17,432,576 (0.85% of 2.06 B)
epochs / optimizer steps	1 / 1,243
batch	1 × 16 grad-accum (effective 16)
learning rate	1e-4, cosine, 3% warmup
precision	bf16, gradient checkpointing, `adamw_torch_fused`
checkpoint selection	`load_best_model_at_end` on `eval_loss`
hardware	1 × RTX 5080 16 GB, 7.09 GB peak VRAM

Final validation loss 0.3289 (from 0.3292 at step 1200 — the curve was flat by the end of the single epoch).

🎯 Accuracy

WER/CER are computed after Vietnamese normalization that expands digits to their spoken form on both sides (334% → ba trăm ba mươi bốn phần trăm) and folds the valid alternative readings (tư/bốn, mốt/một, lăm/năm, ngàn/nghìn, linh/lẻ). Without this, one formatting mismatch can cost 100% WER on a short utterance.

Held-out, speaker-disjoint, unseen — viVoice test

This is the honest number: viVoice test is channel-disjoint from train and no part of it was trained on.

metric	baseline	LoRA	rel. change
WER	5.96%	6.03%	+1.2%
CER	3.03%	2.84%	−6.4%

n = 114 clips (94 in 5–30 s, 20 in 30–60 s). WER is flat and CER improves by 6% — the adapter is getting characters right that the baseline missed while making about as many word-level errors. On a 114-clip set neither delta is significant; read this as "no regression on unseen speakers", not as a gain.

The three external benchmarks

These are no longer zero-shot after fine-tuning — the mixture trains on each of these datasets' training splits. Report them as in-domain adaptation.

dataset	n	baseline WER	LoRA WER	abs	rel
VIVOS (test)	760	7.19%	6.03%	−1.15	−16.0%
Common Voice 17 `vi` (test)	1,274	11.18%	9.87%	−1.31	−11.7%
VLSP 2020 100h (5% held-out)	276	13.16%	9.57%	−3.59	−27.3%

VLSP transcripts are ~96% accurate by the publisher's estimate, so read the delta rather than the absolute. VLSP is also the only one of the three whose held-out slice this project built itself (by transcript hash, so an identical sentence can never land in both train and eval).

Where it regresses — short clips

Scored on 1,250 raw, unmerged viVoice clips:

bucket	n	baseline WER	LoRA WER
0–5 s	892	4.77%	5.13%	worse
5–30 s	358	2.91%	2.78%	better
overall	1,250	3.76%	3.86%

The training mixture is 5–60 s segments, so sub-5-second audio is out of distribution for the adapter in a way it is not for the base model. If your audio is mostly short utterances, benchmark before adopting this adapter.

🖋 Output style — read this before adopting

The adapter learned to condition punctuation on acoustic domain. Measured share of hypotheses containing any of . , ? !:

test set	base model	this adapter
viVoice test (YouTube)	86.8%	99.1%	punctuates more
VIVOS (read speech)	96.4%	0.0%	punctuation gone
Common Voice 17 `vi`	83.5%	1.4%	punctuation gone
VLSP 2020 100h	82.2%	0.0%	punctuation gone

Why: of the four sources, only viVoice ships punctuated transcripts. VIVOS and VLSP do not, and they are 84% of the mixture's clips. The model fit that, and it learned to predict the convention from the audio rather than to punctuate everywhere. This was a known trade-off left in deliberately — punctuation tokens are in-distribution, unlike the all-caps casing problem that was fixed on ingest — but the effect turned out to be near-total on the domains where the training transcripts were bare.

Sentence-initial capitalization moves the same way but far more noisily (viVoice 63.2% → 98.2%, CMV 40.7% → 90.0%, but VIVOS 16.3% → 9.9% and VLSP 35.1% → 18.8%). Do not rely on either signal.

This does not affect the WER/CER tables above — the metric strips punctuation and lowercases before scoring, so the accuracy numbers are unchanged either way. It matters only if you consume the transcript directly.

If you need punctuated output: keep the base model, or run a punctuation restoration model over this adapter's output.

⚠️ Limitations

Vietnamese only. The adapter is trained with language="Vietnamese" in every request; other languages are untested and likely degraded.
Sub-5-second clips regress against the base model (see above).
VIVOS is 59% of the mixture by clip count, so the adapter is pulled toward read-speech acoustics more than the hour counts suggest.
r=16 is sized for 34 hours. If you extend this recipe to a 1,000 h corpus, 17.4 M parameters will underfit — go to r=32/r=64.
Punctuation is domain-conditional — see 🖋 Output style above. This is the limitation most likely to affect you in practice.

📄 License

cc-by-nc-sa-4.0. The base model is Apache-2.0, but viVoice and VIVOS are both CC BY-NC-SA 4.0, and this adapter is a derivative of them — so the non-commercial share-alike terms carry over. Use it for research and personal projects; commercial use would require re-training on permissively licensed audio.