MOSS-TTS-Local v1.5 — Role-Play & Creature-Voice Fine-Tune
A full-parameter fine-tune of
OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5
(4.55 B params, 12-codebook RVQ @ 48 kHz) for
detailed voice-direction following — tone, mood,
emotion, and exaggerated
character / creature voices (Zombie, Gollum-style, Orc, Goblin, Dragon,
Troll, …) — in
English and German (with French/Spanish coverage), using the DramaBox prompt style.
You tell the model what to say (text) and how to say it (instruction), and it performs it.
Two checkpoints in this repo
- root (
./) → creatures_p9 — recommended. Best for role-play, character & creature
voices, emotional direction. Creature performance sits at the ground-truth ceiling.
roleplay_p8_clean/ → roleplay_p8 — the clean-audio model before the creature pass;
slightly stronger neutral studio-English narration, no creature emphasis.
Results
Reference-free proxy metrics (higher = better) vs. ground-truth toplines.
vc_prompt
(
voiceclap-small) measures voice↔direction match
(reliable for English);
vc_text
(
voiceclap-transcriptions) measures
speech↔transcript match (the reliable signal for German/French/Spanish).
| Capability | This model (creatures_p9) | Ground-truth topline |
|---|
Creature/fantasy voices (vc_prompt) | 0.203 | 0.209 |
English role-play direction (vc_prompt) | 0.226 | 0.217 |
German intelligibility (vc_text) | ~0.21 | ~0.22 |
| Runaway generations (>35 s) | 0 / 319 | — |
Creature directions are followed as well as the real creature recordings; English role-play is
above the topline; German intelligibility is near-topline; runaway generations are eliminated.
Usage
1import torch, torchaudio
2from transformers import AutoModel, AutoProcessor
3
4repo = "laion/moss-1.5-roleplay-finetune" # creatures_p9 (root)
5# repo, subfolder = "laion/moss-1.5-roleplay-finetune", "roleplay_p8_clean" # alternative
6
7processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True,
8 codec_weight_dtype="fp32", codec_compute_dtype="bf16")
9processor.audio_tokenizer = processor.audio_tokenizer.to("cuda")
10model = AutoModel.from_pretrained(repo, trust_remote_code=True,
11 dtype=torch.bfloat16, attn_implementation="sdpa").to("cuda").eval()
12
13# Split the prompt at the quote: "how to say it" -> instruction, "what to say" -> text
14conversation = [[processor.build_user_message(
15 instruction="You are a wretched, obsessive creature, your voice a strangled, hissing whisper "
16 "torn between fawning desperation and sudden menace.",
17 text="My precious... yes, yes, we wants it, we needs it.",
18 language="English",
19)]]
20batch = processor(conversation, mode="generation")
21out = model.generate(input_ids=batch["input_ids"].to("cuda"),
22 attention_mask=batch["attention_mask"].to("cuda"),
23 max_new_tokens=1000, do_sample=True,
24 audio_temperature=1.2, audio_top_p=1.0, audio_top_k=25)
25audio = processor.decode(out)[0].audio_codes_list[0] # stereo [2, samples]
26torchaudio.save("out.wav", audio.cpu().float(), processor.model_config.sampling_rate)
Prompting tip: always keep the spoken words in text and the performance direction in
instruction — that matches how the model was trained and gives the highest fidelity. Vivid
second-person directions ("You are a massive ancient dragon, your voice a subterranean rumble…")
work well for creatures.
Training
Full fine-tune (no LoRA) on 2× RTX 3090 via DeepSpeed ZeRO-3 with the optimizer offloaded to CPU.
- Peak LR 2e-5, constant schedule (no warmup/decay); AdamW β=(0.9, 0.95), wd 0.1, bf16.
- Global batch 32 (per-device 4 × grad-accum 4 × 2 GPUs), gradient checkpointing, loss weighting
text:audio = 1:32 across the 12 RVQ heads.
creatures_p9 = phase 8 (2 epochs from base on a clean-audio mix) then 1 further epoch
continuing from it, adding creature + multilingual-emotional data while keeping the phase-8 data
as an anchor against forgetting.
Training data
(An earlier multilingual dataset, TTS-AGI/annotated-audio-tts-training, was used in exploratory
phases but excluded from the final clean-audio model — in-the-wild audio was found to dilute
studio prompt-adherence.)
Full pipeline, experiment log, and ablations:
https://github.com/laion (see the project repo).
License & intended use
Released under CC-BY-NC-4.0 (non-commercial): the base model is Apache-2.0, but part of the
training data (emotional-voice-acting-subset) is CC-BY-NC-4.0, so this derivative inherits a
non-commercial restriction. For research, prototyping, and creative/role-play use.
Limitations
vc_prompt is only a reliable metric for English audio; German/French/Spanish are judged by
vc_text plus human listening.- French/Spanish role-play were validated on multilingual sets, not a dedicated role-play benchmark.
- Neutral studio-English narration is marginally weaker than a narration-specialised checkpoint
(use
roleplay_p8_clean/ if that is your priority).
