A multilingual, multi-speaker text-to-speech model fine-tuned from
unsloth/orpheus-3b-0.1-pretrained
on the fullSunbird/tts
corpus — 20 language configurations and every speaker present in the
dataset.
The model accepts arbitrary text and emits 24 kHz mono speech via the
SNAC audio codec.
Voice selection happens at the prompt level: prepend the chosen
speaker_id followed by ": " to your text, and the model produces
audio in that speaker's voice.
Speaker IDs encode both the source corpus (salt_*, waxal_*, slr32_*,
slr129_*, bateesa_*) and the language. Languages marked with an em dash
in the Speaker IDs column are present in the model's training mix but do
not currently expose individual voice IDs in this checkpoint.
Per-language quality scales with the amount of training data Sunbird
collected for that language; some configs have many more speaker hours
than others. Audition the test split for each language before relying
on a particular speaker — see the discovery snippet below.
TL;DR
python
1# After installing the dependencies (see "Inference" below)2wav = synthesize("Mwattu, oli otya?", speaker_id="salt_lug_0001")# Luganda3wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001")# Swahili4wav = synthesize("Bawo ni, ọrẹ mi?", speaker_id="salt_yor_0001")# Yoruba
The model has no explicit "language" knob — the language identity
travels via the speaker tag, since each salt_<lang>_<NNNN> voice was
recorded in exactly one language.
Discovering speaker IDs
The exact speaker_ids in each config can be enumerated from the dataset:
Speaker IDs follow the pattern salt_<lang>_<NNNN> (e.g.,
salt_lug_0001, salt_ach_0007). Pass any one of them as
speaker_id to either inference function below.
Inference
The model wraps every prompt in a multi-speaker tagged format:
and the model autoregressively emits Llama-3 special tokens followed by
SNAC audio codes that decode to a 24 kHz waveform. Two reference
implementations follow.
Option A — transformers + unsloth (single request)
Best for development, notebook-driven iteration, and small batch sizes.
1import os
2import numpy as np
3import torch
4import soundfile as sf
5from unsloth import FastLanguageModel
6from snac import SNAC
78MODEL_ID ="sunbird/orpheus-3b-tts-multilingual"910# Special tokens — must match the training format11END_OF_TEXT =12800912START_OF_SPEECH =12825713END_OF_SPEECH =12825814START_OF_HUMAN =12825915END_OF_HUMAN =12826016PAD_TOKEN =12826317AUDIO_TOKEN_LO =12826618AUDIO_TOKEN_HI =128266+7*4096# exclusive1920# 1) Load the LM (LoRA already merged into 16-bit weights at training time)21model, tokenizer = FastLanguageModel.from_pretrained(22 model_name = MODEL_ID,23 max_seq_length =4096,24 dtype =None,# auto bf16 / fp1625 load_in_4bit =False,# set True to halve VRAM at slight quality cost26 token = os.environ.get("HF_TOKEN"),27)28FastLanguageModel.for_inference(model)2930# 2) Load SNAC decoder (CPU is fine — frees GPU for the LM)31snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cpu")323334def_redistribute_codes(code_list:list[int])-> torch.Tensor:35 layer_1, layer_2, layer_3 =[],[],[]36for i inrange(len(code_list)//7):37 layer_1.append(code_list[7*i])38 layer_2.append(code_list[7*i +1]-4096)39 layer_3.append(code_list[7*i +2]-2*4096)40 layer_3.append(code_list[7*i +3]-3*4096)41 layer_2.append(code_list[7*i +4]-4*4096)42 layer_3.append(code_list[7*i +5]-5*4096)43 layer_3.append(code_list[7*i +6]-6*4096)44ifnot layer_1:45return torch.zeros(1,1,12000)# ~0.5s silence fallback46 clamp =lambda vals:[max(0,min(4095, v))for v in vals]47 codes =[torch.tensor(clamp(layer_1)).unsqueeze(0),48 torch.tensor(clamp(layer_2)).unsqueeze(0),49 torch.tensor(clamp(layer_3)).unsqueeze(0)]50return snac_model.decode(codes)515253defsynthesize(text:str, speaker_id:str,54*, max_new_tokens:int=1200,55 temperature:float=0.6, top_p:float=0.95,56 repetition_penalty:float=1.1,57 seed:int|None=None)-> np.ndarray:58"""Synthesize speech for `text` in the voice of `speaker_id`.
5960 `speaker_id` must be one of the speakers seen during training,
61 e.g. "salt_lug_0001" (Luganda) or "salt_swa_0003" (Swahili).
62 """63if seed isnotNone:64 torch.manual_seed(seed)6566 tagged =f"{speaker_id}: {text}"67 text_ids = tokenizer(tagged, return_tensors="pt").input_ids
68 soh = torch.tensor([[START_OF_HUMAN]], dtype=torch.int64)69 end = torch.tensor([[END_OF_TEXT, END_OF_HUMAN]], dtype=torch.int64)70 input_ids = torch.cat([soh, text_ids, end], dim=1).to("cuda")71 attention_mask = torch.ones_like(input_ids)7273 generated = model.generate(74 input_ids = input_ids, attention_mask = attention_mask,75 max_new_tokens = max_new_tokens,76 do_sample =True,77 temperature = temperature, top_p = top_p,78 repetition_penalty = repetition_penalty,79 eos_token_id = END_OF_SPEECH, use_cache =True,80)8182# Crop on last SOS, filter to audio token range, redistribute, decode83 sos_indices =(generated == START_OF_SPEECH).nonzero(as_tuple=True)84 cropped = generated[:, sos_indices[1][-1].item()+1:]iflen(sos_indices[1])>0else generated
85 row = cropped[0]86 audio_only = row[(row >= AUDIO_TOKEN_LO)&(row < AUDIO_TOKEN_HI)]87 n =(audio_only.size(0)//7)*788 code_list =[t.item()- AUDIO_TOKEN_LO for t in audio_only[:n]]89 waveform = _redistribute_codes(code_list)90return waveform.detach().squeeze().to("cpu").numpy().astype(np.float32)919293# 3) Use it — pick a speaker per language94wav = synthesize("Mwattu, Mukama yeebazibwe.", speaker_id="salt_lug_0001", seed=42)95sf.write("luganda.wav", wav,24000)9697wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001", seed=42)98sf.write("swahili.wav", wav,24000)
Option B — vllm (high throughput, batched, deployment)
Best for serving traffic. PagedAttention + continuous batching gives
roughly 5–10× faster single-request latency and 10–100× higher
throughput on batched requests vs. the transformers path. Multi-speaker
batching (different speaker_ids in one call) gets the full benefit.
Important: vLLM ships its own torch/transformers and conflicts
with Unsloth's pinned versions. Use a fresh Python environment for
vLLM serving — do not install on top of an Unsloth env.
To recover audio: find the lastSTART_OF_SPEECH (128257) in the
output, take everything after it, drop any token outside the audio
codebook range, group into 7-token frames, undo the per-position offsets,
and feed the three layers to SNAC.decode. Both inference snippets above
implement this end-to-end.
Training details
Setting
Value
Base model
unsloth/orpheus-3b-0.1-pretrained (raw pretrained, not the -ft voice-actor variant)
LoRA merged into 16-bit weights via save_pretrained_merged(save_method="merged_16bit")
The pretrained variant of Orpheus was chosen over the -ft voice-actor
variant because that variant has a strong English-voice-actor prior that
fights low-resource-language fine-tuning.
Data prep summary
Load all 20 configs of Sunbird/tts (get_dataset_config_names)
and concatenate_datasets their train and test splits into one
training set and one held-out evaluation set. No speaker filter.
Tag each row with source = example["speaker_id"] (per-row, not
constant) — the model learns the multi-speaker prompt format
f"{speaker_id}: {text}" across every speaker it sees.
Cast audio to 24 kHz via Audio(sampling_rate=24000).
Drop rows whose tokenised text alone exceeds max_seq_length —
saves expensive SNAC encoding on rows that would be filtered out
downstream.
Encode each remaining audio clip with hubertsiuzdak/snac_24khz →
7 codes per frame, flattened with per-layer offsets
(+128266, +4096, +2·4096, …).
Filter out rows with empty/None codes; drop consecutive duplicate
frames.
Drop rows whose total tokenised length exceeds max_seq_length
(safety net for rows where text fits but text + audio together
overflow the budget).
Evaluation
Quality was evaluated qualitatively on a diverse held-out test sample:
during training, up to 10 utterances are pulled from
ds_test.shuffle(seed=42) covering as many distinct speaker_ids as
possible. Generated audio is saved next to the ground-truth recording
under inference_samples/sample_<idx>_<speaker_id>.wav so each language
/ voice combination can be auditioned individually.
We did not run automated metrics (WER on a downstream STT, MOS
prediction, language-confusion eval, etc.) for this release. Numbers
will be added if/when those become part of the evaluation pipeline.
Important caveat — quality varies by language. The training corpus
is unbalanced across the 20 configs; languages with more speaker hours
in Sunbird/tts get more training signal and produce more natural
speech. Audition the per-language samples before relying on a specific
voice for production traffic.
Intended uses & out-of-scope
Intended:
Multilingual voice synthesis for accessibility, language learning,
human–computer interaction, audio content creation, and downstream
speech research on the 20 covered languages.
A reference checkpoint for the Sunbird/tts → Orpheus-3B multilingual
fine-tuning pipeline; reproducible training recipe in
Orpheus_3B_Sunbird_Multilingual.ipynb.
Out of scope:
Voice impersonation / deception. The model imitates the timbres
of consenting Sunbird voice donors. Do not use the generated audio
to impersonate identifiable real persons or to produce content that
could mislead listeners about who is speaking.
High-stakes decisions. Generated speech may contain pronunciation
errors, prosodic artefacts, or hallucinated phrases — do not deploy
in safety-critical contexts (medical, legal, emergency) without
human review.
Languages outside the 20 configs. The model has no signal for
languages not present in Sunbird/tts; sending German text to any
speaker will produce garbled output, not "German with a Luganda
accent".
Code-switching. Each speaker_id was recorded in a single language;
the model has not seen mixed-language utterances and will likely
produce phonetic artefacts at language boundaries within one prompt.
Cross-language voice transfer. Sending Acholi text to
salt_lug_0001 (a Luganda speaker_id) is undefined behaviour. The
model has no language-conditioning input separate from the speaker
tag, so language identity travels via the speaker_id. Use a speaker
whose salt_<lang>_NNNN prefix matches the language of your text.
Limitations & risks
Quality varies by language. Per-language data volume in
Sunbird/tts is unbalanced. Languages with fewer hours produce
noticeably less natural speech. Run the per-language test-split
audit (script below) before committing to a particular voice.
No language conditioning. There is no language token; the
model relies entirely on the speaker_id to disambiguate. Mismatching
speaker_id and text language is undefined behaviour (see above).
Vocabulary coverage. Limited to the lexicon present in each
config's training subset. Unfamiliar words, code-switching, and
out-of-distribution proper nouns may produce artefacts.
Long utterances. The model was trained on utterances up to ~16 s
of audio (max_seq_length=4096). Generation may degrade or
truncate beyond ~10 s of speech.
Sampling variance. With do_sample=True, identical prompts can
produce noticeably different deliveries between runs. Pass seed=
for reproducibility.
No emotion/style control. Unlike the upstream orpheus-3b-0.1-ft,
this fine-tune was not exposed to in-text emotion tags
(<laugh>, <sigh>, …). Such tags will be tokenised as ordinary
text and produce no special prosodic effect.
Bias. Inherits any biases present in the Sunbird/tts corpus and
in Llama-3's pretraining; we have not audited these systematically
per language.
Audio decoding via SNAC runs on CPU and adds ~50–150 ms per utterance.
License & attribution
This fine-tune is released under Apache-2.0, matching the upstream
unsloth/orpheus-3b-0.1-pretrained
license. It transitively inherits obligations from:
If you only need one specific voice and want a smaller, more focused
checkpoint, see
sunbird/orpheus-3b-tts-salt-lug-0001
— same recipe, scoped to a single Luganda speaker.