Vietnamese Zipformer2 transducer (RNNT) fine-tuned for Actable meeting audio, starting from actableai/zipformer-rnnt-v2 and trained on actableai/vi-meeting-soniox-wer16 with Soniox transcripts as golden labels.


Base	`actableai/zipformer-rnnt-v2` (policy-domain Zipformer, BPE-3000)
Fine-tune data	Meeting segments with WER(Soniox, baseline Qwen) ≤ 16%
Labels	Soniox `text` only (mixed-case; not uppercased)
Params	~70.7 M (transducer; no CTC head in this fine-tune)
Sample rate	16 kHz, 80-dim log-Mel fbank (on-the-fly)

Performance

Meeting dev (Soniox references)

Eval set: session_soniox_dev.jsonl → 5,453 utterances, ~13.8 h
Decoding: greedy search, icefall greedy_search_batch
Text norm for WER: lowercase, strip punctuation / bracket tags, collapse whitespace
Checkpoint: best-valid-loss.pt using model_avg weights

Model	Meeting-dev WER	Errors / words
v2 baseline (`actableai/zipformer-rnnt-v2`)	31.07%	56,369 / 181,450
v3 (this model)	20.71%	37,580 / 181,450
Δ absolute	−10.36	—
Δ relative	−33.3%	—

Measured 2026-07-27 with eval_zipformer_meeting_wer16.py (full meeting dev).

Validation RNNT loss (during fine-tune)

Loss on meeting dev CutSet at the start of each epoch (icefall validation):

Epoch	Valid loss	Simple	Pruned
1	0.4787	0.4529	0.2577
2	0.3846	0.3782	0.1805
3	0.3605	0.3684	0.1725
4	0.3478	0.3633	0.1662
5	0.3433	0.3604	0.1631
6	0.3416	0.3589	0.1622
7	0.3398	0.3578	0.1609
8	0.3385	0.3565	0.1603
9	0.3369	0.3558	0.1590
10 (best)	0.3362	0.3552	0.1586

Best valid loss: 0.3362 @ epoch 10
Best train loss: 0.2452 @ epoch 10
Wall time: ~1 h 38 m (single NVIDIA A100 40GB)

Training details

Data

Split	Segments	Hours	Source
train	22,693	~68.1	`actableai/vi-meeting-soniox-wer16` (= local `session_soniox_train_filtered.jsonl`)
dev	5,453	~13.8	`session_soniox_dev.jsonl` (Soniox labels)

Filter: keep train segments where WER(Soniox ref, baseline Qwen hyp) ≤ 16%
Labels: Soniox field text only — never Qwen hypotheses
Audio: 16 kHz mono session WAVs (full_session.wav + start/end offsets)
Mix / replay: none (use_mux=0) — pure meeting wer16
Utt duration filter: 0.3–30 s

Optimization

Knob	Value
Recipe	icefall Zipformer2 `finetune.py`
Init	load `encoder,encoder_embed,decoder,joiner,simple_am_proj,simple_lm_proj` from v2
CTC	disabled (`use_ctc=0`) — joint CTC FT was unstable (CTC bias grad explosion under fp16)
Optimizer	ScaledAdam + Eden LR schedule
`base_lr`	5e-4
Epochs	10
Batching	`max_duration=300` s / batch, dynamic bucketing
Precision	fp16
Features	on-the-fly 80-dim Kaldi fbank
SpecAugment	on (time-warp factor 80)
MUSAN	off
Speed perturb	off
Seed	42
Hardware	1× A100-SXM4-40GB
Global steps	~8,674

Architecture (unchanged from v2)

Param	Value
`num_encoder_layers`	2,2,3,4,3,2
`encoder_dim`	192,256,384,512,384,256
`downsampling_factor`	1,2,4,8,4,2
`feedforward_dim`	512,768,1024,1536,1024,768
`num_heads`	4,4,4,8,4,4
`cnn_module_kernel`	31,31,15,15,15,31
`encoder_unmasked_dim`	192,192,256,256,256,192
`decoder_dim` / `joiner_dim`	512 / 512
Vocab	BPE-3000 (mixed-case Vietnamese; same as v2)

Files

File	Description
`best-valid-loss.pt`	Best icefall checkpoint (`model` + `model_avg` + train metadata)
`bpe.model`	SentencePiece BPE-3000 (shared with v2)
`tokens.txt`	Token id map for sherpa / icefall
`config.json`	Architecture + training metadata
`train_zipformer_meeting_wer16.sh`	Launch script used for this run
`prep_lhotse_meeting_wer16.py`	Lhotse CutSet prep (Soniox labels)
`eval_zipformer_meeting_wer16.py`	Greedy WER eval helper

Usage (icefall)

python

import torch, sys
sys.path.insert(0, "/path/to/icefall")
sys.path.insert(0, "/path/to/zipformer_work")  # finetune.py / model defs

from finetune import get_model, get_params, add_model_arguments
from beam_search import greedy_search_batch
import argparse, sentencepiece as spm

parser = argparse.ArgumentParser()
add_model_arguments(parser)
params = get_params()
params.update(vars(parser.parse_args([])))
params.encoder_dim = "192,256,384,512,384,256"
params.num_encoder_layers = "2,2,3,4,3,2"
params.downsampling_factor = "1,2,4,8,4,2"
params.feedforward_dim = "512,768,1024,1536,1024,768"
params.num_heads = "4,4,4,8,4,4"
params.cnn_module_kernel = "31,31,15,15,15,31"
params.encoder_unmasked_dim = "192,192,256,256,256,192"
params.decoder_dim = 512
params.joiner_dim = 512
params.causal = False
params.vocab_size = 3000
params.blank_id = 0
params.context_size = 2
params.use_transducer = True
params.use_ctc = False

model = get_model(params)
ckpt = torch.load("best-valid-loss.pt", map_location="cpu")
state = ckpt.get("model_avg") or ckpt["model"]
model.load_state_dict(state, strict=False)
model.eval().cuda()

sp = spm.SentencePieceProcessor()
sp.load("bpe.model")

# feature: (1, T, 80) float32 fbank; feature_lens: (1,) int
with torch.no_grad():
    encoder_out, encoder_out_lens = model.forward_encoder(feature, feature_lens)
    token_ids = greedy_search_batch(model, encoder_out, encoder_out_lens)
text = sp.decode(token_ids[0])
print(text)

Notes / limitations

Optimized for Vietnamese meeting / conversational audio; not re-evaluated on open-domain policy or YouTube suites in this card.
Greedy decoding only in the reported WER; beam search may improve further.
Transcripts are Soniox pseudo-labels (filtered), not human gold.
CTC head from v2 was not fine-tuned / not present in the shipped transducer-only graph.

Changelog

Date	Event
2026-07-27	Fine-tune v2 → meeting wer16, 10 epochs; valid loss 0.479 → 0.336
2026-07-27	Meeting-dev WER 31.07% (v2) → 20.71% (v3); push `actableai/zipformer-rnnt-v3`

actableai/zipformer-rnnt-v3

zipformer-rnnt-v3