Streaming transcription in Chinese and English (including mixed speech)
Turn-state predictions every 80 ms — whether the user is silent, still speaking, finished, or offering a brief backchannel

Built on mistralai/Voxtral-Mini-4B-Realtime-2602, this checkpoint adds an independent turn-prediction head while preserving the original ASR backbone. It is designed for voice assistants that must decide when to wait, respond, ignore a backchannel, or allow interruption.

Model summary


Model ID	`x-square-robot/X2-Turn-4B-0812`
Parameters	~4B
Languages	Chinese, English (mixed)
Frame rate	80 ms
Turn labels	6 classes (see below)
Base model	Voxtral-Mini-4B-Realtime-2602
Paper	arXiv:2608.10878
Code	X-Square-Robot/X2-Turn
License	Apache-2.0 (see LICENSE and NOTICE)

Quick start

Install the inference wrapper from the X2-Turn code repository. No trust_remote_code is required.

bash

git clone https://github.com/X-Square-Robot/X2-Turn.git
python -m pip install -e "./X2-Turn[transformers]"

python

import torch
from transformers import AutoProcessor
from voxtral_realtime.transformers import infer_asr_turn, load_mtp_checkpoint

model_id = "x-square-robot/X2-Turn-4B-0812"

processor = AutoProcessor.from_pretrained(model_id)
model = load_mtp_checkpoint(
    model_id,
    device="cuda",
    dtype=torch.bfloat16,
).eval()

result = infer_asr_turn(model, processor, "/path/to/input.wav")

print("ASR:", result.transcript)
for frame in result.turn_frames:
    print(frame.start_ms, frame.end_ms, frame.label, frame.confidence)

result.transcript — recognized text
result.turn_frames — turn label and confidence for each 80 ms frame

The loader also accepts a local checkpoint directory. For a full command-line example:

bash

python examples/offline_inference.py \
  --model x-square-robot/X2-Turn-4B-0812 \
  --audio /path/to/input.wav \
  --output offline_frames.json

Turn labels

At inference time, the model emits one of six turn states per 80 ms frame. Five of these correspond to the turn states defined and trained in the paper; uncertain is an inference-only label used when the model is not confident that the turn has ended — it was not used during training.

Paper ↔ inference label mapping

Paper label (trained)	Inference label	Meaning
`<\|idle\|>`	`idle`	No useful speech detected
`<\|noidle\|>`	`noidle`	Acoustic activity present, intent not yet clear
`<\|incomplete\|>`	`speaking`	User is still speaking; semantic content is partial
`<\|complete\|>`	`turn_end`	User appears finished; assistant may respond
`<\|backchannel\|>`	`backchannel`	Short acknowledgment (e.g. "嗯", "对", "okay")
— (not trained)	`uncertain`	Low-confidence fallback at inference when the model cannot confidently assign another state

Paper metrics such as ACC_comp and ACC_incomp in Table 1 refer to <\|complete\|> and <\|incomplete\|>, which map to turn_end and speaking respectively at inference time.

These are predictions, not commands. Production systems should smooth across several frames and apply a policy rather than acting on a single frame.

Benchmark results

Evaluation results are reported in X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction (arXiv:2608.10878). This checkpoint is evaluated at streaming delay τ = 480 ms.

Table 1 — Turn state classification on EasyTurn (τ = 480 ms)

	Method	Streaming	ACC_comp ↑	ACC_incomp ↑	ACC_bc ↑	Latency ↓
ZH	SoulX-Duplug	✓	77.67	88.96	—	295 ms
	X2-Turn (Ours)	✓	91.00	93.00	96.00	288 ms
EN	SoulX-Duplug	✓	89.33	79.33	—	205 ms
	X2-Turn (Ours)	✓	92.10	84.60	—	225 ms

Table 2 — Effect of streaming delay τ on EasyTurn

	τ (ms)	ACC_comp ↑	ACC_incomp ↑	Avg. ↑	Latency ↓
ZH	480	91.00	93.00	92.00	288 ms
	400	88.70	94.30	91.50	208 ms
	320	87.33	94.00	90.67	120 ms
EN	480	92.10	84.60	88.49	225 ms
	400	85.20	85.30	85.25	145 ms
	320	82.70	87.60	85.09	65 ms

For the full baseline comparison (including cascaded systems), ASR results (Table 3), and the dual-head architecture overview (Figure 1), see the paper PDF.

Use cases

Recommended for:

Low-latency Mandarin, English, and mixed-language ASR
Voice-assistant response timing and endpointing
Distinguishing real requests from backchannels
Barge-in detection and turn-taking experiments
Controlled research and product evaluation with monitoring

Not recommended for:

Safety-critical decisions without human oversight
Speaker identity or emotion inference
Legal transcription or covert surveillance as the sole basis for action

Realtime serving

Production deployment uses the X2-Turn code repository with a pinned vLLM overlay. Stock vLLM does not emit the custom turn.delta events — follow the vLLM integration guide before serving.

For browser-based visualization of ASR, turn frames, and ACCEPT/REJECT/HOLD/barge-in decisions, see the turn-demo component in the repository.

Architecture

This checkpoint extends Voxtral-Mini-4B-Realtime-2602 with:

Shared backbone and ASR lm_head (stored under base_model.*)
Independent turn head vad_lm_head.weight with full vocabulary

Turn labels map to reserved tokenizer IDs 35–40. The five trained paper labels (idle, noidle, incomplete→speaking, complete→turn_end, backchannel) occupy IDs 35–39; uncertain (ID 40) is reserved for inference-only low-confidence fallback and was not supervised during training.

The canonical model.safetensors uses a single-file layout. The voxtral_realtime.transformers loader creates the VoxtralMTP wrapper before loading both heads. Runtime metadata includes params.json, tekken.json, processor_config.json, and generation_config.json.

Limitations

Accuracy may degrade under noise, reverberation, overlapping speakers, accents, dialects, code-switching, far-field microphones, or packet loss.
Turn predictions can flicker or arrive early/late; temporal smoothing is recommended.
ASR errors and turn errors interact — incomplete text does not always mean an incomplete turn.
Performance may vary across demographic groups, speaking styles, languages, microphones, and environments.
Real-time latency depends on hardware, serving configuration, and policy buffering.

Speech may contain personal, biometric, confidential, or copyrighted information. Obtain appropriate consent, minimize collection and retention, and avoid logging raw audio or transcripts by default.

Citation

If you use this model, please cite:

bibtex

@article{fu2026x2turn,
  title   = {X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction},
  author  = {Kaiqi Fu and Rime Wen and Altman Lin and Shawn Qin and Roy Gan and Hao Wang and Qian Wang},
  journal = {arXiv preprint arXiv:2608.10878},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.10878}
}

License

Model weights and code are released under Apache-2.0. Use is also subject to the Mistral base model terms. See NOTICE for attribution details.