A small, fast Japanese TTS model with a Mamba2 state-space decoder, built on top of
Qwen/Qwen3-TTS-Tokenizer-12Hz.
Kiseki-TTS generates discrete neural audio codec tokens at 12.5 Hz and decodes them to
waveform with the Qwen3 TTS codec. Because the acoustic decoder is a linear-time SSM rather
than a self-attention stack, generation cost is constant per frame — memory does not grow
with utterance length, and there is no KV cache to manage.
Example
Model Details
Model Description
Developed by: telecomadm1145
Model type: Encoder–decoder (Transformer encoder + cross-attention/Mamba2 decoder)
The decoder deliberately has no causal self-attention. Temporal context is carried entirely
by the Mamba2 recurrent state; text conditioning enters through cross-attention whose K/V are
computed once from the encoder and reused for every frame.
Audio tokenization
Property
Value
Frame rate
12.5 Hz (80 ms per frame)
Quantizer layers (Q)
16
Codebook size
2048 per layer
Effective token vocab
2176 (2048 codes + EOS/BOS/PAD, padded to a multiple of 128)
Reserved IDs
EOS = 2048, BOS = 2049, PAD = 2050
Nominal bitrate
16 × 12.5 × log₂(2048) = 2.2 kbps
Max trained length
512 frames ≈ 41 seconds
Depth modelling (MTP head). Each frame's 16 codebook layers are predicted by a shared
"multi-token prediction" head rather than 16 separate decoder passes. Layer q sees the
decoder hidden state plus the exclusive prefix sum of the embeddings of layers 0 … q-1:
where Block is a small RMSNorm → SwiGLU(×2) → RMSNorm residual body shared across all 16
layers. This means one trunk evaluation per frame and 16 cheap head evaluations, instead of
16 full autoregressive steps.
Why it's fast
1. 12.5 Hz is the headline number.
One second of speech is 12.5 decoder steps. Codecs running at 50 Hz or 75 Hz need 4–6×
more autoregressive steps for the same audio. Concretely:
Audio duration
Decoder trunk steps
Codebook head evals
1 s
12.5
200
5 s
63
1,000
10 s
125
2,000
30 s
375
6,000
A 10-second utterance is 125 recurrent steps. For comparison, a token-level LLM TTS at
50 Hz × 8 codebooks would be pushing ~500 trunk steps for the same clip.
2. O(1) state, not O(T) cache.
The Mamba2 decoder carries a fixed (32 heads × 128 state × 64 dim) tensor plus a 3-frame
conv window per layer. Generating 40 seconds costs exactly as much per step as generating
1 second — no attention matrix, no KV cache reallocation, no quadratic blowup. Long-form
synthesis degrades gracefully instead of falling off a memory cliff.
3. Cross-attention K/V is computed once.
Encoder output is projected to per-layer K/V a single time during prefill. Every subsequent
frame does one small Q·Kᵀ against a fixed-length text sequence.
4. A shallow decoder.
Only 6 decoder layers sit in the autoregressive loop. The 12-layer encoder runs exactly once,
fully parallel over the input text.
5. The depth loop is cheap.
The 16 codebook layers are resolved sequentially (layer q conditions on layers <q), but
each step is one ×2 SwiGLU block at d=1024 — small enough that batch-1 generation is
memory-bandwidth-bound rather than compute-bound.
