A LoRA adapter for MOSS-VoiceGenerator (1.7B) trained with Group Relative Policy Optimization (GRPO) to improve speaker similarity, emotion expression, and speech intelligibility.
1import requests, base64, io, soundfile as sf
23response = requests.post("http://localhost:30000/generate", json={4"text":"${instruction:happiness}Hello, how are you today?",5"audio_data":["speaker_reference.wav"],# absolute path to ref audio6"sampling_params":{7"temperature":1.0,8"top_p":0.8,9"top_k":50,10"repetition_penalty":1.1,11"max_new_tokens":4096,12},13"stream":False,14})1516# Decode base64 WAV response17audio_bytes = base64.b64decode(response.json()["text"])18wav, sr = sf.read(io.BytesIO(audio_bytes))# 24kHz19sf.write("output.wav", wav, sr)
Or use the included CLI tool:
bash
1python serve_sglang.py generate \2 --text "Hello, how are you today?"\3 --emotion happiness \4 --ref-audio speaker_reference.wav \5 --output output.wav
How GRPO Works
Group Relative Policy Optimization (GRPO) is a reinforcement learning method that directly optimizes generation quality:
Generate: For each text prompt, generate G=4 audio completions with the current model
Score: Rate each completion with three reward models:
Speaker similarity (ECAPA-TDNN): cosine similarity between generated and reference speaker embeddings
Emotion match (Voice-OpenCLAP): cosine similarity between audio embedding and emotion text
Intelligibility (Parakeet ASR): Word Error Rate of transcribed audio vs target text
Advantage: Normalize rewards within each group: A_i = (R_i - mean) / std
Train: Update model with policy gradient: L = mean(A_i * CE_loss(completion_i))
This trains the model to produce audio that better matches the target speaker, conveys the right emotion, and maintains intelligibility.
Important Notes
Always use merge_and_unload() before generation. The PeftModel wrapper breaks MOSS's multi-head architecture and produces garbled audio.
The LoRA only modifies model.language_model (Qwen3 backbone). Audio embedding layers and output heads are unchanged.
Audio is generated at 24kHz with 16 VQ codebooks using a delay pattern.
Training
This section describes how to replicate GRPO training using the included grpo_train_v6.py and grpo_rewards.py scripts. The pipeline uses SGLang for fast generation, enabling semi-on-policy GRPO on a single 8-GPU node.
Dependencies
bash
1# Core2pip install torch>=2.3 transformers peft accelerate safetensors
34# Audio processing5pip install librosa soundfile numpy jiwer
67# Reward models8pip install speechbrain # ECAPA-TDNN speaker verification9pip install openai-whisper # Whisper ASR (fallback)10pip install nemo_toolkit[asr]# Parakeet ASR (preferred)1112# CLAP model (clone separately)13# See: https://huggingface.co/laion/voice-openclap-poc1415# SGLang (OpenMOSS fork with MOSS TTS support)16pip install"sglang[all]" --find-links https://github.com/OpenMOSS/sglang
17# Or install from source:18# git clone https://github.com/OpenMOSS/sglang && cd sglang && pip install -e ".[all]"1920# Data21pip install datasets # HuggingFace datasets for voice-acting-prompts
Architecture Overview
The GRPO training pipeline uses a semi-on-policy design with pipeline overlap across 8 GPUs:
GPU 0: Training (forward/backward/optimizer) — base model + LoRA
GPUs 1-6: SGLang server with data-parallel-size=6 (continuous batching)
GPU 7: Reward models (ECAPA-TDNN, Voice-OpenCLAP, Parakeet ASR)
The key idea is that generation (the bottleneck in GRPO) runs on a dedicated SGLang server with 6-way data parallelism, while training and reward scoring happen on separate GPUs. This enables pipeline overlap: while the training GPU processes batch N (scoring rewards + computing gradients), the SGLang server is already generating completions for batch N+1.
Semi-on-policy: LoRA weights are synced from the training model to the SGLang server every --sync-every steps (default 5). Between syncs, the SGLang server generates using slightly stale weights — a practical trade-off that avoids the costly restart-per-step approach while keeping the policy reasonably fresh.
Weight sync works by:
Saving the current LoRA adapter to disk
Merging it into a fused model checkpoint (only regenerating the modified safetensors shard)
Triggering SGLang to reload the updated weights
Data Pipeline
Dataset: voice-acting-prompts — a large collection of expressive text prompts with emotion labels.
IMPORTANT: English-only filtering is required. The dataset is approximately 78% non-English (German, French, etc.). The _is_english function in grpo_train_v6.py filters prompts by checking the ratio of ASCII characters to total characters (threshold: 80% ASCII). Without this filter, the model will train on non-English text and WER rewards become meaningless.
IMPORTANT: Always shuffle the streaming dataset. The dataset shards are not randomly ordered — early shards are heavily German. Without shuffling, the first several hundred steps would contain almost exclusively non-English prompts that pass through even the ASCII filter. The training script uses dataset.shuffle(seed=42) to ensure a uniform language distribution across training.
Reference speakers: Emolia dataset speaker clusters (3000 clusters of expressive speech). Each training step randomly samples a speaker cluster and a random utterance within that cluster as the reference audio for voice cloning.
Prompt processing:
Stream and shuffle the dataset
Filter to English-only using ASCII character ratio
Clean text (remove quotes, stage directions, normalize whitespace)
Pair each prompt with a randomly sampled emolia speaker reference
Format as MOSS 4-message ICL: [empty_user, ref_assistant, target_user, gen_assistant]
Reward Models
Three reward models run on GPU 7, wrapped by grpo_rewards.py:
Model
What it measures
Output range
Class
ECAPA-TDNN (speechbrain)
Speaker similarity between generated and reference audio
[-1, 1] cosine
SpeakerReward
Voice-OpenCLAP
Emotion match: audio-text cosine similarity to emotion description
[-1, 1] cosine
CLAPReward
Voice-OpenCLAP (quality)
Audio quality: similarity to "High Quality Recording, fluid pleasant performance"
[-1, 1] cosine
CLAPReward.score_quality()
Parakeet TDT 0.6B (or Whisper fallback)
Intelligibility: Word Error Rate of transcribed audio vs target text
[0, inf) WER
ASRReward
All audio is resampled to 16kHz before reward scoring.
Reward Formula
Individual rewards are z-normalized using running baseline statistics, then combined with a multiplicative WER penalty:
w_spk, w_clap, w_qual — reward component weights (must sum to 1.0)
beta — WER penalty strength (default: 10.0)
WER — word error rate from ASR transcription
The exponential WER penalty ensures that unintelligible speech receives near-zero total reward regardless of how well it matches the speaker or emotion. At WER=0, the penalty is 1.0 (no effect); at WER=0.3, the penalty is ~0.05.
Reward weight configurations used:
v1 (default): w_spk=0.6, w_clap=0.4, w_qual=0.0 — speaker-heavy, no quality term
v7: w_spk=0.5, w_clap=0.4, w_qual=0.1 — balanced with quality bonus
GRPO Algorithm Details
For each training step:
Sample a batch of B=8 prompts from the dataset
Generate G=4 completions per prompt using SGLang (total 32 audio samples)
Score each completion with all three reward models
Combine rewards using the formula above
Compute advantages within each group: A_i = (R_i - mean(R_group)) / std(R_group)
Train using advantage-weighted cross-entropy loss on the audio tokens
The loss is computed as:
L = mean(A_i * CE_loss(completion_i))
where CE_loss uses channelwise loss weighting (--channelwise-loss-weight "1,32" means text head weight=1, total audio heads weight=32 spread across 16 codebook heads).
Key Training Parameters
Parameter
Default
Description
--lora-init
(required)
Path to initial LoRA adapter or HF repo
--output-dir
output/grpo_v6
Directory for checkpoints and logs
--batch-size
8
Prompts per training step
--group-size
4
Completions generated per prompt (G)
--lr
5e-5
Learning rate
--max-steps
1600
Total training steps
--sync-every
5
Steps between LoRA-to-SGLang weight syncs
--save-every
200
Steps between checkpoint saves
--channelwise-loss-weight
"1,32"
Text head vs total audio weight
--w-speaker
0.5
Speaker similarity reward weight
--w-clap
0.4
Emotion match reward weight
--w-quality
0.1
Audio quality reward weight
--beta-wer
10.0
WER penalty strength
--train-device
cuda:0
GPU for training
--sglang-gpus
1,2,3,4,5,6
GPUs for SGLang server (DP)
--reward-device
cuda:7
GPU for reward models
--lr-schedule
constant
LR schedule: constant, cosine, or linear
--warmup-steps
0
Linear warmup steps
--lr-min
0.0
Minimum LR for cosine/linear decay
--seed
42
Random seed
--resume-step
0
Resume from this step number
LoRA Configuration
LoRA is applied only to model.language_model — the Qwen3 backbone that handles sequence modeling. The 16 audio embedding layers (emb_ext) and 17 output heads (lm_heads) are frozen and unchanged. This is critical because:
The audio codebook embeddings map discrete VQ codes to continuous representations — modifying them would break the learned codebook alignment
The output heads project back to codebook logits — these must remain calibrated to the frozen embeddings
The Qwen3 backbone is where high-level decisions about what to generate are made, making it the right target for RL fine-tuning
LoRA config: rank=8, alpha=16, dropout=0.0, applied to all linear layers in the Qwen3 model (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), using TaskType.FEATURE_EXTRACTION.
Replicating v7 Training
v7 continued from the v1 checkpoint (500 steps) with adjusted reward weights. To replicate:
bash
1# Requires: 8x GPUs (H100 80GB recommended), ~2-3 hours for 500 steps2# The script will automatically:3# 1. Load base model + LoRA on GPU 04# 2. Launch SGLang server on GPUs 1-65# 3. Load reward models on GPU 76# 4. Stream and filter the voice-acting-prompts dataset78python grpo_train_v6.py \9 --lora-init output/grpo/final \10 --output-dir output/grpo_v7 \11 --max-steps 500\12 --save-every 100\13 --sync-every 5\14 --w-speaker 0.5 --w-clap 0.4 --w-quality 0.1\15 --lr 5e-5
To start from scratch (no prior LoRA, fresh rank-8 adapter):
Environment variables (optional, for custom paths):
bash
1exportMVG_DIR=/path/to/MOSS-VoiceGenerator # Base model2exportCODEC_DIR=/path/to/MOSS-Audio-Tokenizer # Audio codec3exportEMOLIA_DIR=/path/to/emolia/cluster_samples # Reference speakers4exportCLAP_DIR=/path/to/voice-openclap-poc # CLAP model
Training Tips
Monitor WER closely. If WER rises above ~0.3, the model is producing unintelligible speech. Consider increasing --beta-wer or reducing the learning rate.
Speaker similarity and emotion match trade off. Increasing --w-speaker improves voice cloning fidelity but may reduce emotional expressiveness, and vice versa.
Checkpoints every 100-200 steps are recommended. Peak metrics often occur mid-training (e.g., v7 peaked at step 258 for speaker sim, step 472 for CLAP) and final checkpoints may not be optimal.
--sync-every 5 is a good default. Lower values (1-2) keep the policy more on-policy but increase overhead from weight sync. Higher values (10+) risk the SGLang server generating with stale weights, reducing training signal quality.
Channelwise loss weight "1,32" means the text token head has weight 1 and all 16 audio codebook heads share a total weight of 32 (2 per head). This upweights audio quality relative to text token prediction.
