A frame-synchronous full-duplex speech LoRA for google/gemma-4-E2B-it. It listens and
"thinks" on the same clock as the incoming audio, so turn-taking, barge-in (interrupting the
model mid-reply), and multi-turn conversation are handled natively by the model — not by an
external VAD or scripting layer.
The model emits text frame-by-frame; speech is produced by any downstream TTS. This keeps the
adapter tiny (24M trainable params, ~96MB) while the base model does the heavy lifting.
Frame-synchronous full-duplex: the model decides wait/speak on every 80ms frame; barge-in flips it back mid-reply
Turn-taking — stays silent while you speak, opens up when you finish
Barge-in — stop mid-reply the moment you interrupt
Multi-turn — holds a conversation across many turns without going silent
Tool calling — emits structured tool calls and speaks the result
Text input — accepts typed messages mid-conversation
All in one adapter, bilingual (Chinese / English).
Frame-synchronous: the core difference from base Gemma
Base Gemma is turn-based: you send a whole message, it generates a whole reply, done. There is
no notion of time — nothing happens "while you are still talking."
This adapter makes the same model frame-synchronous. Time is sliced into 80ms frames, and
on every single frame the model does two things at once: it reads the latest audio and decides
what to emit. Each frame is 2 audio tokens + 1 text token, and that one text token is a live
decision:
time → frame 1 frame 2 frame 3 … frame k frame k+1 …
audio [you ...... speaking .............. pause] (silence)
model <wait> <wait> <wait> … <wait> "Sure," "here's"…
└─ stays quiet while you talk ─┘ └─ opens up on its own ─┘
Emitting <wait> = "keep listening." Emitting a word = "I'm speaking now."
Turn-taking is just: when do I switch from <wait> to words?
Barge-in is: your voice returns mid-reply → the model flips back to <wait> within a few frames.
Multi-turn is: keep doing this correctly as the conversation state evolves.
Because every decision lives on a shared clock with the audio, these behaviors are the model's
own, learned end-to-end — not bolted on by a separate turn-detector.
How it works (in one idea)
The LoRA sits only on the language model's attention/MLP projections; the audio encoder and
embeddings stay frozen — we teach behavior, not perception.
The key finding behind this release: opening up, shutting up (on barge-in), and staying in a
multi-turn conversation are the same problem — the "act now" signal is a tiny fraction of the
supervised positions and gets drowned out by all the <wait> frames. Amplifying the training loss
on those few decisive positions cures all three at once, with plain supervised fine-tuning.
Performance
Bare model, all runtime heuristics off (no VAD, no nudge) — this is the model on its own:
Capability
Metric
Result
Opening (standard)
replies after you finish
11/12
Opening (rambling questions)
"
12/12
Barge-in
chars still spoken after interrupt (median)
7
Multi-turn
5-turn survival (turn 1 / overall)
8/8 · 5/8 stable
Text channel
replies to typed input
24/24
Tool calling
correct call + spoken summary
11/12
With a light runtime layer (energy-gated barge, silence-freeze), the live experience is smoother
still — see the demo repo.
Usage
This is a PEFT LoRA adapter for google/gemma-4-E2B-it.
The adapter runs frame-synchronously (80ms frames), so it needs a streaming driver to feed
audio and drain text/tool events. A companion demo repo with a ready-to-run web app (microphone UI
Semantics are bounded by a 2B base — great at conversational behavior, not a knowledge oracle.
Barge-in isn't instant — the model shortens its tail to a few characters; a runtime energy
gate handles hard cut-off. Native stop is strong but not zero-latency.
Text-out only — you bring your own TTS.
zh / en only.
License & attribution
License: Apache-2.0 (inherited from the gemma-4-E2B-it base).
