This is the fine-tuned conversational LLM of a DuplexCascade fork: a VAD-free cascaded ASR–LLM–TTS pipeline where a single LLM drives full-duplex spoken dialogue by emitting special tokens that control turn-taking, interruptions, backchannels and thinking pauses — without any voice-activity detection.

The model is a QLoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507 on synthetic pt-BR duplex micro-turn data, trained with the weighted/masked cross-entropy objective from the DuplexCascade paper (§4.1). This repo also hosts DuplexCascade-Distill, a second model that learns the same protocol without the capability regression of plain SFT — see DuplexCascade-Distill below.

Files

Path	Description
`models/adapter/`	QLoRA adapter of the SFT baseline, V0 (LoRA weights + tokenizer + `training_args.bin`) — resume fine-tuning from here
`models/export/gguf/DuplexCascade-PT-q4_k_m.gguf`	SFT baseline GGUF q4_k_m quantization for llama.cpp serving
`data/dialogues_combined.jsonl`	Synthetic pt-BR dialogue corpus (short + long replies) used for training
`models/distill/adapter/`	QLoRA adapter of DuplexCascade-Distill (resumable fine-tune state)
`models/distill/export/gguf/DuplexCascade-Distill-q4_k_m.gguf`	Distill model GGUF q4_k_m for llama.cpp serving
`models/distill/export/gguf/Modelfile`	Ollama Modelfile: serving recipe + why the distill model exists and its results
`data/distill/teacher_duplex_train.jsonl`	Distill fine-tuning data: 10k pt-BR dialogues, assistant content regenerated by the frozen base under the duplex trigger, converted to duplex micro-turns
`data/distill/anchor_prompts.jsonl`	Capability-anchor prompts (function calling / general QA / reasoning) for the distill KL term

Builders and training code live in the source repos (duplex_cascade, duplex_cascade_distill).

How it works

The model is trained to consume interleaved micro-turns supervised with special tokens:

Token	Meaning
<	user is speaking
`<	user finish speaking
`<	no voice
`<	user is thinking
`<	user interruption
`<	user backchannel
`<	system backchannel

Loss is computed only on system micro-turns, and each special token carries a task-specific loss weight (paper §4.1): <|user finish speaking|> ×10, <|user interruption|> ×5, <|system backchannel|> ×3, <|user backchannel|> ×2, <|user is speaking|>/<|user is thinking|> ×1.

At inference the LLM emits these tokens to negotiate the conversation flow in real time — no VAD, no fixed turn-taking.

Training

Base model: Qwen/Qwen3-4B-Instruct-2507 (bf16, dense)
Method: 4-bit NF4 QLoRA, r=16, α=16, dropout 0, on all linear modules (q/k/v/o/gate/up/down), plus modules_to_save=["embed_tokens", "lm_head"] so the new special-token embeddings are trained
Objective: custom DuplexSFTTrainer implementing weighted/masked cross-entropy over system micro-turns only
Data: synthetic pt-BR chat dialogues generated with Gemma 4 E4B (llama.cpp), converted to duplex micro-turns with probabilistic interruptions/backchannels/thinking pauses and variable system chunk lengths (10–48 tokens)
Stages: initial fine-tune (~2k steps) → continuation fine-tune (~1k steps) from the merged model on a short+long-response mixture
Context: 2048 tokens, ChatML format
Stack: Unsloth + PEFT + Hugging Face TRL, 16 GB GPU

Usage

llama.cpp (recommended)

bash

llama-server -m DuplexCascade-PT-q4_k_m.gguf \
  --port 8080 -sp \
  --jinja --chat-template chat_template.jinja

-sp (special-token emission) is required — without it the duplex special tokens are silently dropped from completions.

Transformers + PEFT

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
model = PeftModel.from_pretrained(base, "lumierenoir/DuplexCascade-PT-BR-V0/models/adapter")
tokenizer = AutoTokenizer.from_pretrained("lumierenoir/DuplexCascade-PT-BR-V0/models/adapter")

Full speech-to-speech demo

The full stack (llama.cpp + faster-whisper STT + pocket-tts TTS + web UI) is orchestrated by run_services.sh in the source project.

DuplexCascade-Distill

models/distill/… + data/distill/… hold the second model in this repo.

Why it exists

Plain SFT on the duplex protocol regresses general capabilities: the V0 baseline above learns turn-taking (98.0% tag accuracy) but its function calling collapses to 55.6% and general pt-BR instruction-following to 41.7% — textbook catastrophic forgetting under off-policy SFT.

DuplexCascade-Distill replaces the hard content fit with a self-distillation objective (SDFT, arXiv:2601.19897, adapted without RL or on-policy rollouts): the assistant content is generated by the frozen base model itself prompted with a duplex trigger; during fine-tuning the 7 special tags are learned by weighted CE while every other token position is pulled toward the frozen base's own distribution via per-token forward-KL (disable_adapters() teacher pass, offline top-k=32 cache). The KL term pins "everything else" to the pretrained model — that is the capability-retention mechanism.

Found results

Model	Function calling (18 pt-BR tasks)	General (12 pt-BR tasks)	Tag accuracy	KL drift
base	100%	100%	0.0%	0.51*
distill (this repo)	100%	100%	96.4%	1.07
sft (V0 baseline)	55.6%	41.7%	98.0%	4.76

* top-k truncation floor of a perfectly-matched model (reference).

Standard benchmarks (2000-example subsets): distill stays within ~1–2 pp of base on HellaSwag (0.566 vs 0.584), ARC-Challenge (0.563 vs 0.584) and MMLU (0.698 vs 0.705); the SFT baseline loses 4.5 pp on MMLU (0.660). Extra benchmarks: distill even beats base on TruthfulQA mc2 (0.649 vs 0.626) and IFEval loose (0.762 vs 0.730); its single honest regression is GSM8K long-form CoT (0.777 vs 0.890). The SFT baseline drops on all of them.

Full tables and reproduction commands: eval/RESULTS.md.

Usage

bash

llama-server -m DuplexCascade-Distill-q4_k_m.gguf \
  --port 8080 -sp \
  --repeat-penalty 1.15

-sp is required so duplex special tokens appear in completions.
To resume distill training: load models/distill/adapter/ onto Qwen/Qwen3-4B-Instruct-2507 with PEFT and continue with training/train_distill.py.

Caveats

Fine-tuned for Brazilian Portuguese spoken dialogue; other languages will degrade.
Trained on a limited synthetic corpus — factual accuracy and long-form coherence are secondary to turn-taking behavior.
Served via a cascaded ASR–LLM–TTS pipeline, so end-to-end latency is dominated by the ASR/TTS backends.
DuplexCascade-Distill: one measured regression vs base is GSM8K long-form multi-step reasoning (−11.3 pp); a lower KL weight (λ) or EMA-teacher variants are the natural follow-ups.

Known issues

Distill: reply opens by continuing the user's utterance (mitigated 2026-08 in the github)

The distill model occasionally opens its reply with the continuation of the user's own sentence instead of a fresh answer — e.g. user: "Qual é a capital do Brasil?" → assistant: ", a capital do Brasil é Brasília." (leading comma

echo of the user's words).

Why: the teacher (frozen base) sometimes autocompleted the user turn when generating the assistant content under the v1 "Responda de forma curta" trigger, and the content forward-KL anchors that mode into the trained model. The v2 trigger at inference reduces it but does not remove it.

Current mitigation: the bridge strips leading whitespace + continuation punctuation from every micro-turn before TTS/transcript/history (_strip_leading_continuation in distill/DuplexCascade/server.py).

Planned fix (future): regenerate the teacher training data with the v2 trigger ("responda de forma natural e completa … que responda de fato ao que o usuário perguntou … não se repita") and retrain — see distill/SPEC.md §6 R0 — so the continuation artifact is gone from the model natively instead of being stripped at inference.

License & provenance

Vendored inference base: MIT (sbintuitions/DuplexCascade)
Training data: generated by open models (Gemma); fine to distribute
Whisper (MIT), pocket-tts (CC BY 4.0), Common Voice pt (CC0)

Citation

Based on:

DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR–LLM–TTS Pipeline and Micro-Turn Optimization — Jianing Yang, Yusuke Fujita, Yui Sudo (sbintuitions).