Streaming Japanese ASR that also marks fillers and repairs, trained on
CSJ + CEJC by Shinya Fujie (Fujie Lab, Chiba Institute of Technology).
This is the composite tokens (ア+F / ン+D) variant. The auxiliary information is fused into the token itself: a filler kana is recognized as ア+F, a repair kana as ン+D. Strip the +F / +D suffix to recover the plain transcript.
Two sibling models encode the same information differently — see
the comparison below:
Runs with current ESPnet via
fujielab-asr — no old-commit
checkout needed.
pip install "fujielab-asr>=0.2.0"
python
1import numpy as np, soundfile as sf
2from fujielab.asr.espnet_ext.espnet2.bin.asr_transducer_inference_cbs import Speech2Text
34s2t = Speech2Text.from_pretrained(5"fujie/espnet_asr_csj_cejc_pron_comp_cbs_transducer_120300_hop132",6 streaming=True, lm_weight=0.0,7 beam_size=20, beam_search_config=dict(search_type="maes"),8)910audio, fs = sf.read("utterance.wav")# 16 kHz mono11chunk =int(16000*0.1)# 100 ms12n, text =len(audio),""13for i inrange(0, n, chunk):14 c = audio[i:i + chunk]15 is_final = i + chunk >= n # flag the last chunk even when it is full16iflen(c)< chunk:17 c = np.pad(c,(0, chunk -len(c)))18 res = s2t.hypotheses_to_results(s2t.streaming_decode(c, is_final=is_final))19if res:20 text = res[0][0]21print(text)
Only search_type="maes" is exercised by these models.
CER is computed with all auxiliary information stripped from both reference
and hypothesis, so the three encodings are measured against the same target.
Scoring the markup itself would penalise comp and span for text the
aux-head model never has to emit, which is what made an earlier comparison
look like a 2–3 point gap when the real gap is under half a point.
CER %
F: P / R / F1
D: P / R / F1
this model
16.25
82.9 / 81.2 / 82.0
56.4 / 38.1 / 45.5
By corpus: CEJC 22.89 % / CSJ 5.50 % CER.
F = filler (フィラー), D = repair / disfluent restart (言い直し).
Detection counts as correct when the predicted span overlaps the reference span
of the same type within ±1 token.
The three encodings compared
All three models share the encoder geometry, training data and schedule, and
differ only in how the auxiliary information is encoded:
encoding
CER %
F F1
D F1
aux (separate head)
16.01
81.9
44.4
comp (ア+F)
16.25
82.0
45.5
span (<F> … </F>)
16.41
81.5
43.9
A paired bootstrap over utterances calls every pairwise CER gap significant,
but the comp vs span gap flips sign between the 14-epoch and 50-epoch
checkpoints (+0.14 → −0.17), so run-to-run variance exceeds it: treat those two
as indistinguishable. aux is ahead of both at either training length, though
by only 0.23–0.40 points. comp has the best D detection at both lengths.
D recall is 35–38 % across all three — the weak point of every model here.
Predictions are reasonably precise (55–60 %) but miss most repairs.
Training
Encoder: Contextual Block Conformer, 12 blocks, d=256, 4 heads,
macaron + conv module (kernel 15). Left context 12 / main block 3 /
look-ahead 0 — i.e. no look-ahead, so the encoder is causal at block
granularity. block_size 15, hop_size 3.
Decoder: RNN-Transducer, single-layer LSTM predictor (hidden 512),
joint space 640.
Frontend: 80-dim log-mel, hop_length 132 samples (8.25 ms) at 16 kHz,
×4 subsampling → one encoder frame per 33 ms.
Data: CSJ + CEJC, kana (pronunciation) tokens. Fillers are identified
with the CSJ-aligned criterion, which also treats short vowel fillers
(えー / まあ / あー / ん) as F — a harder detection target than the
UniDic-based criterion, and one that costs about 8 points of F F1 while
leaving CER unchanged.
Schedule: 50 epochs × 1400 steps, Adam lr 0.0035, warmup 2500,
effective batch 32 M bins (16 M × 2 × H100).
14 epochs was substantially undertrained for this setup: extending to 50 moved
CER by 2.4–2.9 points and D F1 by 7–12 points, far more than the choice of
auxiliary encoding does.
Note on positional encoding and long audio
The encoder adds an absolute sinusoidal positional encoding indexed from
the start of the stream, and its self-attention is the plain (not relative)
variant. Training utterances here run to 19.9 s at most (498 encoder frames),
so feeding a long continuous stream pushes the position index far outside the
trained range. Segment the audio (e.g. by VAD) and let the recognizer reset per
segment rather than streaming for minutes on end.
Citing ESPnet
bibtex
1@inproceedings{watanabe2018espnet,
2 author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi
3 and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann
4 and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala
5 and Tsubasa Ochiai},
6 title={{ESPnet}: End-to-End Speech Processing Toolkit},
7 year={2018},
8 booktitle={Proceedings of Interspeech},
9 pages={2207--2211},
10 doi={10.21437/Interspeech.2018-1456}
11}
