You can deploy this model and start earning money today!

fujie/
espnet_asr_csj_cejc_pron_aux_cbs_ctc_120300_hop132

Name: espnet_asr_csj_cejc_pron_aux_cbs_ctc_120300_hop132
Availability: InStock

espnet

audio

automatic-speech-recognition

csj

cejc

cc-by-4.0

Views

No views yet

`fujie/espnet_asr_csj_cejc_pron_aux_cbs_ctc_120300_hop132`

Streaming Japanese ASR that also marks fillers and repairs, trained on CSJ + CEJC by Shinya Fujie (Fujie Lab, Chiba Institute of Technology).

CTC-only head — no decoder, greedy decoding. This is the multitask CTC (separate N/F/D head) variant. The transcript is plain kana. The auxiliary information is a parallel label sequence (N / F / D per recognized token) from a separate head on the encoder output, so the text itself carries no markup.

Sibling models: the same three encodings with a CTC head (aux / comp / span) and with an RNN-Transducer head.

Usage

pip install "fujielab-asr>=0.2.0"

python

import numpy as np, soundfile as sf
from fujielab.asr.espnet_ext.espnet2.bin.asr_multitask_ctc_inference_cbs import (
    Speech2TextMultitaskCTC,
)

s2t = Speech2TextMultitaskCTC.from_pretrained(
    "fujie/espnet_asr_csj_cejc_pron_aux_cbs_ctc_120300_hop132", streaming=True
)

audio, fs = sf.read("utterance.wav")   # 16 kHz mono
chunk = int(16000 * 0.1)               # 100 ms
n, result = len(audio), None
for i in range(0, n, chunk):
    c = audio[i:i + chunk]
    is_final = i + chunk >= n          # flag the last chunk even when it is full
    if len(c) < chunk:
        c = np.pad(c, (0, chunk - len(c)))
    r = s2t.streaming_decode(c, is_final=is_final)
    if r:
        result = r[0]

# tokens and aux_labels are aligned 1:1
print(" ".join(f"{t}[{a}]" if a != "N" else t
               for t, a in zip(result.tokens, result.aux_labels)))

Evaluation

17 evaluation sets (CEJC 14 + CSJ 3), 23,500 utterances, greedy CTC decoding with valid.cer_ctc.ave_10best.pth.

CER is computed with all auxiliary information stripped from both reference and hypothesis, so every encoding is measured against the same target.

	CER %	F: P / R / F1	D: P / R / F1
this model	16.90	80.3 / 82.8 / 81.6	55.7 / 35.2 / 43.1

By corpus: CEJC 23.75 % / CSJ 5.83 % CER.

F = filler (フィラー), D = repair / disfluent restart (言い直し). A detection counts as correct when the predicted span overlaps a reference span of the same type within ±1 token.

CTC vs. RNN-Transducer

A sibling model with the identical encoder, data, effective batch (32 M), schedule (50 × 1400 steps, lr 0.0035, warmup 2500) and evaluation exists with an RNN-Transducer head — only the head differs, so the gap is attributable to it:

head	CER %
CTC (this model)	16.90
RNN-T (`..._aux_cbs_transducer_120300_hop132`)	16.01
difference	+0.89

The RNN-T head wins on all three encodings (+0.33 to +1.39 points, each significant under a paired bootstrap over utterances), but the margin varies fourfold by encoding, so there is no single "RNN-T is N points better" figure.

Auxiliary-label quality barely depends on the head: across all six models F F1 spans 80.4–82.0 and D F1 spans 41.6–45.5. What does change is the precision/recall balance — the CTC models lean toward recall, the RNN-T models are balanced — which is the more useful basis for choosing between them.

Pick CTC when you want a smaller, faster, decoder-free model and can accept ~0.3–1.4 points more CER; pick RNN-T for the best transcript.

Training

Encoder: Contextual Block Conformer, 12 blocks, d=256, 4 heads, macaron + conv module (kernel 15). Left context 12 / main block 3 / look-ahead 0, block_size 15, hop_size 3 — causal at block granularity.
Head: CTC only (ctc_weight 1.0, no decoder), plus a per-token auxiliary head (N/F/D) on the encoder output.
Frontend: 80-dim log-mel, hop_length 132 samples (8.25 ms) at 16 kHz, ×4 subsampling → one encoder frame per 33 ms.
Data: CSJ + CEJC, kana (pronunciation) tokens, CSJ-aligned filler criterion (short vowel fillers えー / まあ / あー / ん also count as F).
Schedule: 50 epochs × 1400 steps, Adam lr 0.0035, warmup 2500, effective batch 32 M bins (16 M × 2 × H100). Peak memory 55 GB.

Note on positional encoding and long audio

The encoder adds an absolute sinusoidal positional encoding indexed from the start of the stream, and uses plain (not relative) self-attention. Training utterances reach 19.9 s at most (498 encoder frames), so a long continuous stream pushes the position index outside the trained range. Segment the audio (e.g. by VAD) and let the recognizer reset per segment.

Citing ESPnet

bibtex

@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi
          and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann
          and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala
          and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456}
}