Streaming Japanese ASR that also marks fillers and repairs, trained on
CSJ + CEJC by Shinya Fujie (Fujie Lab, Chiba Institute of Technology).
This is the multitask Transducer (separate N/F/D head) variant. The transcript is plain kana. The auxiliary information is a parallel label sequence (one of N / F / D per recognized token), predicted by a separate head, so the text itself stays free of markup.
Two sibling models encode the same information differently — see
the comparison below:
Runs with current ESPnet via
fujielab-asr — no old-commit
checkout needed.
pip install "fujielab-asr>=0.2.0"
python
1import numpy as np, soundfile as sf
2from fujielab.asr.espnet_ext.espnet2.bin.asr_multitask_transducer_inference_cbs import(3 Speech2TextMultitaskTransducer,4)56s2t = Speech2TextMultitaskTransducer.from_pretrained(7"fujie/espnet_asr_csj_cejc_pron_aux_cbs_transducer_120300_hop132",8 beam_size=20, beam_search_config=dict(search_type="maes"),9)1011audio, fs = sf.read("utterance.wav")# 16 kHz mono12chunk =int(16000*0.1)# 100 ms13n, result =len(audio),None14for i inrange(0, n, chunk):15 c = audio[i:i + chunk]16 is_final = i + chunk >= n # flag the last chunk even when it is full17iflen(c)< chunk:18 c = np.pad(c,(0, chunk -len(c)))19 r = s2t.streaming_decode(c, is_final=is_final)20if r:21 result = r[0]2223# tokens and aux_labels are aligned 1:124print(" ".join(f"{t}[{a}]"if a !="N"else t
25for t, a inzip(result.tokens, result.aux_labels)))
Only search_type="maes" is exercised by these models.
Streaming vs. offline auxiliary labels
The auxiliary head sits on the joint network's pre-output hidden state, so a
label only exists at a specific (frame, predictor-state) node of the Transducer
lattice. The offline research recipe picks that node by re-encoding the whole
utterance and forced-aligning the final hypothesis — a second pass over
complete audio, which a streaming recognizer cannot perform.
fujielab-asr instead reads the head at the node the beam search actually
emitted the token at. That needs no second pass, so auxiliary labels are
available on every 100 ms chunk rather than only at end of utterance.
Measured agreement between the two protocols on 297 held-out CEJC utterances
(4,619 tokens), holding the hypothesis and the encoder output fixed so that
only the choice of node varies:
agreement
all tokens
99.50 %
tokens where either protocol says F or D
90.17 %
Emission frames coincide exactly for 54 % of tokens and land within one frame
(33 ms) for 92 %. So the streaming labels track the reported offline numbers
closely, but they are not bit-identical to them — worth knowing if you are
reproducing the paper figures.
CER is computed with all auxiliary information stripped from both reference
and hypothesis, so the three encodings are measured against the same target.
Scoring the markup itself would penalise comp and span for text the
aux-head model never has to emit, which is what made an earlier comparison
look like a 2–3 point gap when the real gap is under half a point.
CER %
F: P / R / F1
D: P / R / F1
this model
16.01
82.6 / 81.1 / 81.9
59.5 / 35.4 / 44.4
By corpus: CEJC 22.60 % / CSJ 5.36 % CER.
F = filler (フィラー), D = repair / disfluent restart (言い直し).
Detection counts as correct when the predicted span overlaps the reference span
of the same type within ±1 token.
The three encodings compared
All three models share the encoder geometry, training data and schedule, and
differ only in how the auxiliary information is encoded:
encoding
CER %
F F1
D F1
aux (separate head)
16.01
81.9
44.4
comp (ア+F)
16.25
82.0
45.5
span (<F> … </F>)
16.41
81.5
43.9
A paired bootstrap over utterances calls every pairwise CER gap significant,
but the comp vs span gap flips sign between the 14-epoch and 50-epoch
checkpoints (+0.14 → −0.17), so run-to-run variance exceeds it: treat those two
as indistinguishable. aux is ahead of both at either training length, though
by only 0.23–0.40 points. comp has the best D detection at both lengths.
D recall is 35–38 % across all three — the weak point of every model here.
Predictions are reasonably precise (55–60 %) but miss most repairs.
Training
Encoder: Contextual Block Conformer, 12 blocks, d=256, 4 heads,
macaron + conv module (kernel 15). Left context 12 / main block 3 /
look-ahead 0 — i.e. no look-ahead, so the encoder is causal at block
granularity. block_size 15, hop_size 3.
Decoder: RNN-Transducer, single-layer LSTM predictor (hidden 512),
joint space 640.
Frontend: 80-dim log-mel, hop_length 132 samples (8.25 ms) at 16 kHz,
×4 subsampling → one encoder frame per 33 ms.
Data: CSJ + CEJC, kana (pronunciation) tokens. Fillers are identified
with the CSJ-aligned criterion, which also treats short vowel fillers
(えー / まあ / あー / ん) as F — a harder detection target than the
UniDic-based criterion, and one that costs about 8 points of F F1 while
leaving CER unchanged.
Schedule: 50 epochs × 1400 steps, Adam lr 0.0035, warmup 2500,
effective batch 32 M bins (16 M × 2 × H100).
14 epochs was substantially undertrained for this setup: extending to 50 moved
CER by 2.4–2.9 points and D F1 by 7–12 points, far more than the choice of
auxiliary encoding does.
Note on positional encoding and long audio
The encoder adds an absolute sinusoidal positional encoding indexed from
the start of the stream, and its self-attention is the plain (not relative)
variant. Training utterances here run to 19.9 s at most (498 encoder frames),
so feeding a long continuous stream pushes the position index far outside the
trained range. Segment the audio (e.g. by VAD) and let the recognizer reset per
segment rather than streaming for minutes on end.
Citing ESPnet
bibtex
1@inproceedings{watanabe2018espnet,
2 author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi
3 and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann
4 and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala
5 and Tsubasa Ochiai},
6 title={{ESPnet}: End-to-End Speech Processing Toolkit},
7 year={2018},
8 booktitle={Proceedings of Interspeech},
9 pages={2207--2211},
10 doi={10.21437/Interspeech.2018-1456}
11}
