⚠️ Do not merge this adapter into the base weights
merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint"
script will destroy the model irrecoverably. This is not a performance caveat. Read this
before you write a deployment script.
Why
This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23
target modules. In this architecture those output heads are weight-tied to the input
embeddings: tie_weights() sets
audio_lm_heads[i].weight IS audio_embeddings[i].weight # the same tensor, not a copy
text_lm_head.weight IS transformer.embed_tokens.weight
They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head
weight, it writes that delta straight into the embedding table at the same time. The model
then reads its own inputs through a matrix that has been shifted by an output-side correction.
Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage
is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the
original values are gone.
Verify it yourself in three lines
Do not take our word for it:
python
1m = base.model ifhasattr(base,"model")else base
2print(m.audio_lm_heads[0].weight.data_ptr()== m.audio_embeddings[0].weight.data_ptr())3# True -> same storage, merging corrupts the embeddings
What to do instead
Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:
python
1from peft import PeftModel
23model = PeftModel.from_pretrained(base,"<this repo>", adapter_name="a").to(dev).eval()4# do NOT call model.merge_and_unload()56defset_weight(model, name, w):7"""Scale one named adapter's contribution. alpha/r is its own base scaling."""8for module in model.modules():9 scaling =getattr(module,"scaling",None)10ifisinstance(scaling,dict)and name in scaling:11ifnothasattr(module,"_base_scaling"):12 module._base_scaling ={}13 module._base_scaling.setdefault(name, scaling[name])14 scaling[name]= module._base_scaling[name]*float(w)1516set_weight(model,"a",1.0)17model.base_model.set_adapter(["a"])# several adapters can be active at once
This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r),
which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same
arithmetic, in a different order. You give up a small amount of inference speed and you keep the
ability to change the weight, stack several adapters, or turn one off. Nothing about the sound
changes.
If you are stacking adapters
Set each one's scaling separately and activate them together with
model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a
deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add
adapters deliberately and measure.
If you maintain code that merges
A regex over module names is not enough — the reliable test is identity of storage. Group the
modules by weight.data_ptr() and refuse to merge into any group with more than one member.
lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.
500 low-rank adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, one per synthetic voice profile. Each adapter makes the frozen base model speak as one specific, invented character voice — a "grave documentary narrator", an "anxious teenage hacker" — and ships with the reference clip and the profile card that define that character. Every one of them is rank 4, alpha 8, dropout 0.05 over the same 23 target modules, weighs about 34 MB, and beats the frozen base model on that voice's own held-out clips. None of them is a clone of a real person: every voice profile is synthetic, invented from a design spec and then generated.
If you have never seen this project before, read §1 to §3 and then run quickstart.py.
Not buried at the bottom, because these change how you should use the release. Full detail in §12.
No human has listened to any of this in a controlled study. Every quality number here is held-out language-modelling loss on MOSS audio codes, plus ECAPA speaker cosine. Those are proxies for "sounds like the voice", not measurements of it.
You need the reference clip as well as the weights. The base model is reference-conditioned. An adapter with no reference gives you a random speaker wearing the adapter's colouring — there is no useful "no reference" mode. The right clip ships at voices/<voice>/reference.wav.
8 of the 500 adapters are below the measured saturation point and 6 of those are drawn from voices the generator never cloned reliably. They are flagged and named (§8), not hidden.
Do not read a voice's gender out of caption_general. The corpus renders the gender and background-noise axes of that prose field with inverted polarity, and about 35 % of training instructions carried the buggy prose verbatim. Use card_gender from profile.json / the manifest (§12).
Relationship to TTS-AGI/moss-voice-profile-loras
There is an earlier, smaller LoRA repo, and it is not superseded silently, so here is what it actually is.
TTS-AGI/moss-voice-profile-loras (public, 168 files) is the 10-voice pilot and rank-ablation study that this release was designed from. It holds ten voices, each with a shipped adapter and six ablation arms (ranks/{r4,r8,r16}/{stage1,stage2}) plus ablation/ result tables. That sweep is where rank 4 was chosen; it is why every adapter here is rank 4 and not rank 16.
All ten of its voices — anime_088, emolia_c0542, emolia_c1682, emolia_c1699, emolia_c2570, k10_age3_bg1, k325_age3_bg1, k395_age3_bg1, k91_age5_bg0, mediathek_0184 — are present in this release, and the pilot repo's own README says so, describing this build as the one whose voices "will replace these same ten".
TTS-AGI/moss-voice-profile-loras
this repo
purpose
rank/stage ablation, pilot
production release
voices
10
500
ranks
4, 8, 16 (60 ablation arms)
4 only
ships reference clip
no
yes
ships profile card
no
yes
manifest
ablation/*.csv
manifest.parquet, 56 columns
Keep the pilot repo if you want the rank ablation or the two-stage arms. For actually generating a voice, use this one.
1. What these are
A voice profile here is a stable, invented speaker identity — not a real person and not a recording of one. Each profile started as a written design spec (name, tagline, gender, age, accent, register, timbre sliders, casting notes) and a single reference clip. That profile was then used to generate a large corpus of takes across emotion, delivery, character and edge-case prompts — roughly 38,000 candidate takes per voice. The adapter shipped here is fitted on the subset of those takes that most convincingly sound like the profile, as scored by an ECAPA speaker embedder against the reference.
The result is a 34 MB file that, attached to the 4.55 B base model, holds that one identity across the whole expressive range the corpus covers: emotions, deliveries, vocal bursts, English and German. Nothing in this release is trained on a real speaker's voice, and there is no real identity to recover from these weights.
These are stage-1 identity adapters (stage = "stage1" on all 500 rows). They are complete and usable as they are.
2. What is in the repo
README.md this file
MANIFEST_COLUMNS.md the column dictionary, also inlined at §10
manifest.parquet one row per voice, 500 rows x 56 columns
manifest.json the same rows as JSON
RELEASE.json build summary + the downstream consumer contract
quickstart.py runnable end-to-end example
voices/<voice>/
adapter_model.safetensors the LoRA weights, peft format (~34.4 MB)
adapter_config.json the peft config
reference.wav the voice's reference clip -- REQUIRED at generation time
profile.json the voice's profile card (name, tagline, casting, tags)
voice.json this voice's manifest row, standalone
Choosing a voice. Use the profile-card columns in the manifest — voice_name, tagline, card_gender, card_age, card_accent, card_register, card_tags. These come from the design spec each voice was generated from. Do not use the corpus's caption_general prose for this (§12).
pandas gotcha:rank is also a DataFrame method. Write m["rank"], never m.rank. The same applies to any column whose name collides with a method.
If you are consuming this release from code, read manifest.parquet and treat voice as the key. Every row already points at that voice's final adapter — the choice between a retrain and its incumbent has already been made, so there is nothing to resolve:
python
1m = pd.read_parquet("manifest.parquet").set_index("voice")2rel = m.loc["emolia_c0019","adapter_dir"]# "voices/emolia_c0019" -- relative, use this3was_new =bool(m.loc["emolia_c0019","retrained"])# did this release retrain it?4why = m.loc["emolia_c0019","retrain_status"]# shipped_retrain / rejected_worse / ...
The *_abs path columns and release_root record absolute paths on the machine that built the release and are provenance only; on the Hub, use adapter_dir. Never load from adapter_src — it points into build trees that are not part of this release.
3. How to use one
quickstart.py in this repo is the code below, runnable:
Budget roughly 20 GB of VRAM for the base model plus the codec.
Three things actually trip people up:
(a) You need the reference clip as well as the adapter. Covered above; quickstart.py defaults to the shipped one, so in practice you do not have to find it.
(b) PeftModel.active_adapter is a plain instance attribute, not a property. It is assigned once in PeftModel.__init__ and is not kept in sync by everything that changes adapters — on the class itself it is simply None. active_adapters (plural) IS a property and does query the model. Read the plural one, or read peft_config directly, and never assume the singular attribute is current after juggling adapters.
(c) peft 0.20.0's offline path mangles subfolder. With HF_HUB_OFFLINE=1, passing subfolder= to PeftModel.from_pretrained puts the subfolder into the filename and passes it again as a kwarg, so it looks for <voice>/<voice>/adapter_model.safetensors and raises LocalEntryNotFoundError. Resolve to a plain local directory first, as below, and it works either way.
python
1import numpy as np, soundfile as sf, torch
2from huggingface_hub import snapshot_download
3from peft import PeftModel
4from transformers import AutoModel, AutoProcessor
56BASE ="laion/moss-tts-local-transformer-4.55b-voice-acting-v2"7CODEC ="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"8REPO ="laion/moss-voice-profile-loras-500"9VOICE ="emolia_c0019"# any directory name under voices/1011# Pull just the one voice (~34 MB + a reference clip), not all 17 GB.12root = snapshot_download(REPO, allow_patterns=[f"voices/{VOICE}/*","manifest.parquet"])13adir =f"{root}/voices/{VOICE}"1415proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True, codec_path=CODEC)16proc.audio_tokenizer = proc.audio_tokenizer.to("cuda").eval()1718# "some weights of MossTTSLocalModel were not initialised" warnings about audio_lm_heads /19# text_lm_head are benign -- those tensors are weight-tied to the embeddings, not missing.20model = AutoModel.from_pretrained(BASE, trust_remote_code=True,21 dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()2223model = PeftModel.from_pretrained(model, adir).eval()# a plain local directory24name = model.active_adapters[0]# PLURAL: the property (see (b))25print("rank:", model.peft_config[name].r,"alpha:", model.peft_config[name].lora_alpha)2627conv =[[proc.build_user_message(28 text="I have read the file, and that is exactly what worries me.",29 instruction="A warm, unhurried voice, speaking just above a murmur.",30 language="English",# "English" or "German"31 reference=[f"{adir}/reference.wav"],# ships with the adapter -- do not omit32 tokens=12)]]# a length HINT in words, not a limit33batch = proc(conv, mode="generation")3435out = model.generate(input_ids=batch["input_ids"].cuda(),36 attention_mask=batch["attention_mask"].cuda(),37 max_new_frames=400,# the real ceiling: 400 frames ~ 32 s at 12.5 fps38 do_sample=True,39 text_temperature=0.7, text_top_k=50, text_top_p=1.0,40 audio_temperature=1.0, audio_top_p=0.95, audio_top_k=30,41 audio_repetition_penalty=1.1)4243msg = proc.decode(out)[0]44ifnot msg.audio_codes_list:# a NORMAL, silent failure mode of this model -- always check45raise SystemExit("empty decode; retry with another seed")46w = msg.audio_codes_list[0].cpu().float().numpy()47sf.write("out.wav", np.ascontiguousarray(w.mean(0)if w.ndim >1else w),48000)
To merge an adapter permanently: model = model.merge_and_unload(). These adapters were evaluated at scale 1.0 only; no other merge scale was measured, and nothing here was measured with two voice adapters attached at once.
4. What the adapter actually touches
Read straight out of the shipped adapter_model.safetensors, not out of the config. 536 tensors, 268 adapted modules, 8,589,312 trainable parameters, identical in shape across all 500 voices.
audio_lm_heads.0–11 — one head per MOSS codebook, each (4, 2560) → (1024, 4)
12
172,032
2.00 %
local_transformer.h.0 — the talker head, c_attn, c_proj, fc_in, fc_out
4
159,744
1.86 %
total
268
8,589,312
100 %
This is the single most useful fact for deciding how to use these adapters. About 96 % of the capacity sits in the semantic/prosodic transformer and only about 3.9 % in the acoustic path — and of the local transformer's stack, only layer 0 is adapted at all. These adapters overwhelmingly change how the model decides what the voice does — phrasing, timing, prosodic shape, register — and only marginally retune the codebook decoding on the way out. If you expected a timbre patch, this is not one; the timbre still comes substantially from the reference clip, which is why §3(a) matters as much as it does.
All 500 adapters share this exact structure: rank = 4, lora_alpha = 8 and n_target_modules = 23 on every manifest row, and all 500 adapter_sha256 values are distinct.
5. How they were trained
Per voice:
Take the voice's ~38,000 generated candidate takes and drop the empty ones, anything under 0.5 s, and every clip belonging to a held-out group (40 groups per voice, stratified by block and language, fixed seed). Held-out audio never enters training.
Select the training pool by ECAPA speaker similarity against the voice's reference (§6, §7).
Encode the pool to MOSS audio codes — 12 codebooks × 1024, 12.5 frames/s exactly (hop 3,840 at 48 kHz).
Train rank 4 / alpha 8 / dropout 0.05 on the 23 target modules for 6 epochs, checkpointing and evaluating on the held-out clips after every epoch.
Ship the epoch with the lowest held-out loss. If no epoch beat the frozen base model, ship nothing. That never happened: all 500 beat their base.
Captions are resampled every epoch from the stored measurements, so a clip is not welded to one phrasing of its description.
Why best-epoch selection rather than a fixed epoch count. Across the 499 runs of the original build the winning epoch was ep2 ×90, ep3 ×297, ep4 ×99, ep5 ×13 — there is no constant that is right for all voices. Every one of those runs overfits: the best epoch was never the last (499/499). But the rise after the minimum is strictly monotone in only 472/499 (94.6 %), so "it gets worse after the best epoch" is reliable while "it gets worse at every subsequent epoch" is not — an early-stop on the first uptick would misfire on about 5 % of voices. Picking the minimum over all six is what makes the 100 % beat-base rate real rather than lucky.
Across the shipped 500 adapters (which for 119 voices means the retrain's epoch, not the original run's) the chosen epoch is ep2 ×54, ep3 ×331, ep4 ×101, ep5 ×13, plus stage2 ×1 for emolia_c1699.
Cost. Summing gpu_h over the shipped adapters gives 378.7 GPU-hours (repair build 329.3, 3,000-clip retrain 47.3, floor-capped retrain 0.8, reference study 1.2). That column records only the stage that produced the shipped weights; counting the discarded challengers too, the build reports roughly 355 GPU-h for the original 499 plus 51 GPU-h for the retrain pass, ≈406 GPU-h in total. Encoding time is not included in either figure.
Provenance of the 500. 499 come from the identity-repair pipeline (src_tree = repair, lora3k or lora3k_fc); the 500th, emolia_c1699, is the original single-voice reference study and used a different, two-stage recipe (§8).
6. The two thresholds, and what they do and do not mean
Two numbers govern which clips a voice was trained on. They are both cosine similarities between ECAPA speaker embeddings, on a scale internal to this project, and they are easy to confuse.
threshold
name
what it does
0.40
identity floor
Below this, a take is treated as a failed clone — the generator produced somebody else.
0.60
MIN_SPK_TRAIN
The training-pool threshold of the original recipe: "train on the takes above 0.60". A purity-seeking cut, not a correctness cut.
The original recipe took every take at spk_sim ≥ 0.60, capped at the 6,000 highest. If fewer than 800 cleared, it relaxed to that voice's own top 800 and flagged the voice relaxed; below 200 it refused to train at all. The retrain recipe replaced this with the voice's top 3,000 takes with a hard 0.40 floor (§7).
These are conventions of this project, not established perceptual facts. The project's own report is explicit: "The 0.40 floor is this project's ECAPA scale, not an established perceptual fact: an independent WavLM-tbr embedder puts 75.4 % of the selected failures above its own same-speaker threshold (0.472)." Two embedders disagree about three quarters of the rejected population.
Three different speaker embedders appear around this project and none of their scales are interchangeable. Worth spelling out, because a threshold copied across them is silently wrong:
embedder
dim
used for
a threshold on that scale
ECAPA (speechbrain/spkrec-ecapa-voxceleb)
—
every spk_sim in this release
0.40 floor, 0.60 training cut
Orange/Speaker-wavLM-tbr
128
the independent cross-check quoted above
its own same-speaker point, 0.472
Orange/Speaker-wavLM-id
250
a different, ear-calibrated task elsewhere
0.80
The 0.80 in particular belongs to -id and must not be applied to -tbr or to ECAPA. A cosine is only meaningful in the embedding it was computed in.
min_spk_effective in the manifest is the number that actually matters per voice: the lowest similarity that really entered that voice's pool, whatever rule produced it. Across the release: 378 voices realise ≥ 0.60, 119 land in [0.40, 0.60), 2 fall below 0.40 (emolia_c1340 at 0.388, emolia_c2142 at 0.387 — both kept pre-floor incumbents, see §8), and 1 is null (emolia_c1699, whose pool was not similarity-filtered).
One more caveat from the same report, about what hard identity filtering costs: "Identity is bought, not free: the reference run paid −11 % target strength and −14 % blend for its identity gain." Selecting for clips that sound like the profile is not the same as selecting for clips that are expressive or well articulated.
7. Why 3,000 clips
Over the 499 adapters of the original build, gain over base correlates with pool size at r = +0.532 and with pool purity (mean speaker similarity) at only r = +0.234. And it saturates. Measured on the pre-retrain state of all 500 voices:
pool clips
n voices
median gain
0–999
60
0.1063
1,000–1,999
35
0.1288
2,000–2,999
28
0.1541
3,000–4,999
54
0.1630
5,000–6,000
322
0.1626
>6,000
1
0.2253
3,000–4,999 and 5,000–6,000 are indistinguishable. Going past ~3,000 clips buys nothing; staying under it costs a lot. The 123 voices trained on fewer than 3,000 clips had a median gain of 0.1221 against 0.1627 for the rest — about 25 % less improvement.
So those 123 voices, and only those, were retrained on the voice's top 3,000 takes by speaker similarity with a hard 0.40 floor — not on a flat 0.60 cut, because 124 of the 500 voices cannot reach even 800 clips at 0.60, which is exactly why they were under-trained in the first place. The other 377 were left alone: they are at or above saturation, and retraining them would have burned GPU-hours to move a number that has stopped moving.
Never ship a worse adapter. Every challenger was written to a separate tree and evaluated on the same held-out clips as the incumbent — the validation container was copied verbatim, not rebuilt, which is the only way two losses are comparable. The lower loss wins.
outcome
voices
3,000-clip retrain shipped
115
originals-only floor-capped variant shipped
4
all challengers rejected (incumbent shipped)
4
retrain did not finish (incumbent shipped)
0
not a target (already saturated, untouched)
377
119 voices shipped a new adapter, 4 kept their incumbent, and 0 shipped worse — that last figure guaranteed by construction, since a losing challenger is discarded. Median gain across those 123: 0.1221 → 0.1427, a median per-voice improvement of +0.0219 (median +0.0227 among the winners). Voices below saturation: 123 → 8.
Pool composition beats pool size
Of the 123 retrain targets, 117 could fill 3,000 clips from their own original takes above the floor; 6 could not and had their pools topped up with audio regenerated by the identity-repair pipeline.
retrain pool
improved on incumbent
all-original (117 voices)
115 / 117
contains regenerated audio (6 voices)
0 / 6
Every pool containing regenerated audio regressed, without exception. All-original pools overwhelmingly improved, but not universally: emolia_c2305 and k492_age2_bg1 had clean 3,000-clip retrains that still lost to their incumbents, and shipped the incumbent. No shipped adapter in this release contains a single regenerated clip — pool_n_repaired = 0 and used_repaired = false on all 500 rows.
Gain by pool size across the shipped release now looks like this:
pool clips
n voices
median gain
0–999
2
0.0820
1,000–1,999
4
0.1050
2,000–2,999
2
0.1554
3,000–4,999
169
0.1492
5,000–6,000
322
0.1626
>6,000
1
0.2253
8. Which voices are weak, and why
Honesty matters more here than a tidy release. 8 of the 500 adapters are still below the 3,000-clip saturation point and should be expected to clone their profile less reliably than the rest. They are flagged below_saturation = true in the manifest and named here:
voice
pool clips
gain
status
sub-floor
emolia_c1340
800
0.0751
rejected_worse
yes
emolia_c2142
800
0.0888
rejected_worse
yes
emolia_c1070
1,096
0.0931
shipped_floorcap
yes
emolia_c0241
1,532
0.1049
shipped_floorcap
yes
emolia_c0697
1,861
0.1052
shipped_floorcap
yes
emolia_c0243
1,909
0.1220
shipped_floorcap
yes
k492_age2_bg1
2,881
0.1543
rejected_worse
no
emolia_c2305
2,933
0.1566
rejected_worse
no
The six sub-floor voices
The most informative result in the release. Six voices could not supply 3,000 original takes above the 0.40 identity floor — their generator failed to sound like the profile most of the time. Their incumbent adapters had been trained on 800 clips selected by dropping the threshold until 800 existed, which for emolia_c1340 and emolia_c2142 meant reaching down to similarity 0.388 and 0.387, i.e. below the identity floor, including clips that are arguably a different speaker. Two repairs were tried, and they came out opposite ways.
Attempt 1 — more clips, from regenerated audio. Failed, six times out of six. Topping the pool up with identity-repair regenerations reached a full 3,000 clips at a far higher effective similarity threshold for every one of them. Every one came out worse than the 800-clip incumbent, losing roughly a third to a half of its gain. The regenerated audio is measurably degraded on content and delivery — for emolia_c1070, word error rate +96 %, naturalness −23 %, burst blend −38 % — and the validation clips are original audio. An adapter fitted on regenerated audio predicts regenerated audio.
Attempt 2 — fewer clips, but all original and all above the floor. Won, up to a point.
voice
incumbent: clips / gain
3k regen: clips / gain
floor-capped: clips / gain
shipped
final gain
emolia_c0241
800 / 0.0915
3,000 (2,975 regen) / 0.0448
1,532 / 0.1049
floorcap
0.1049
emolia_c0243
800 / 0.1040
3,000 (2,929 regen) / 0.0672
1,909 / 0.1220
floorcap
0.1220
emolia_c0697
800 / 0.0894
3,000 (2,957 regen) / 0.0540
1,861 / 0.1052
floorcap
0.1052
emolia_c1070
800 / 0.0849
3,000 (2,865 regen) / 0.0512
1,096 / 0.0931
floorcap
0.0931
emolia_c1340
800 / 0.0751
3,000 (2,944 regen) / 0.0322
546 / 0.0681
incumbent
0.0751
emolia_c2142
800 / 0.0888
3,000 (2,861 regen) / 0.0559
632 / 0.0834
incumbent
0.0888
What this shows. The saturation curve in §7 is not about clip count as such — it is about the count of clips drawn from the distribution you are evaluated on. Three thousand regenerated clips lose to 800 originals. But cleanliness stops paying once the pool gets small enough: emolia_c1340 (546 clean clips) and emolia_c2142 (632) both lost to their 800-clip incumbents even though those incumbents contain sub-floor audio, while emolia_c1070won on 1,096. The crossover therefore sits between 632 and 1,096 clips: above it a smaller clean pool wins, below it there is simply not enough data and the larger contaminated pool is the better bet. That is considerably more useful than "3,000 is the magic number", and it is the opposite of what a naive reading of the saturation table predicts.
The two voices whose pools reach below 0.40 are also the two with the lowest gains in the entire release — at least consistent with treating the floor as a training constraint and not only a reporting one.
All six remain the weakest voices here and stay flagged sub_floor_voice = true. Improved or not, each is fitted on a pool far under saturation, drawn from a voice the generator never cloned reliably. Expect less consistent identity from all six, and prefer other voices where you have the choice.
emolia_c1699 is not like the others
emolia_c1699 is the original single-voice reference study the whole project was built from. Its adapter comes from a different recipe — a two-stage curriculum over 18,823 unfiltered clips (plus a 1,580-clip stage-2 sharpening pass), selected by a rank sweep (4 / 8 / 16) and a stage sweep rather than the single-stage similarity-filtered recipe used for the other 499. It is rank 4 like the rest and its held-out loss is measured on the same kind of split, but it is not a comparable data point about the recipe — it has the highest gain in the release (0.2253) partly because it is the only voice trained this way. recipe = reference_study_2stage marks it, and pool_mean_spk / min_spk_effective are null for it because its pool was never similarity-filtered.
9. profile.json, reference.wav, and which gender field to trust
reference.wav is the voice's conditioning clip — the same clip the profile was defined by and the corpus was generated from, at the enhancement variant the profile card rates best by DNSMOS (reference_variant in the manifest says which of orig / sidon / cbx; across the release: 173 / 156 / 171). The base model is reference-conditioned, so this file is not optional. These 500 files are the only published copy of the profiles' reference clips — see the note below.
profile.json is the voice's design spec — the document the voice was generated from, not a measurement of the audio that came out:
json
1{2"cid":"emolia_c0019",3"name":"Grave Documentary Narrator",4"tagline":"A weightful and measured voice that conveys historical gravity and calm authority.",5"gender":"Male",6"age":"Late 40s to 50s",7"language":"English",8"accent":"Standard American",9"register":"Mid-range baritone with a calm, downward inflection",10"timbre_profile":{"metallic":1,"throat_guttural":2,"falsetto":0,11"chest_voice":4,"roughness":2,"brightness":3},12"distinctive_features":"Precise enunciation combined with a somber, grounded resonance…",13"emotional_range":"Primarily serious and informative with a steady, cautionary undertone.",14"casting":{"classic_fantasy":{"role":"Royal Archivist","direction":"…"},15"sci_fi":{…},"mystery_horror":{…},"contemporary":{…}},16"tags":["english","standard american","male","middle-aged","narrator", …]17}
The casting blocks are genre-specific role + direction pairs and make good ready-made instruction= strings.
card_gender is the field to trust. It is the profile card's gender, i.e. the intended design, and it is not derived from the corpus's buggy caption_general prose (§12). Anyone wanting a voice's gender should read card_gender (or profile.json's gender) and nothing else. Note it states intent, not a measurement of the produced audio. Across the release: Male 242, Female 236, Androgynous 21, null 1.
One voice, k532_age3_bg1, has a stub profile card with no descriptive fields, so voice_name, tagline and the card_* columns are null there. Its adapter and reference clip are unaffected. card_language is additionally null for 115 voices whose cards did not record one.
9b. The speaker name — name, and a third gender field
Every voice now also carries a given name: one unique, ordinary first name per profile, added
so that a model can be conditioned on "speak as Katrin" instead of on a reference clip. It is in
manifest.parquet / manifest.json (name, name_gender_class, name_confidence), in every
voices/<voice>/voice.json, and standalone in the new top-level names.csv / names.json.
profile.json is deliberately left byte-untouched.
Names are decorative labels for conditioning. They are not the character names — those stay
where they were, in voice_name (manifest, voice.json) and in name (profile.json, unchanged).
name_gender_class is measured, and it is not the same thing as card_gender.
field
what it describes
who produced it
card_gender
the reference clip
a VLM writing the character card during voice selection
name_gender_class
the 40,256 takes this profile actually generated
VoiceNet vn_GEND_reg, thresholded on the corpus's own bimodal distribution
A voice is male when at least 85 % of its takes score vn_GEND_reg >= 3.2103 (the 50/50
boundary of a 2-component Gaussian mixture fitted to 1,003,974 clips: modes at 1.587 and 4.867),
female when at most 15 % do, and uncertain in between — 209 male, 170 female, 121 uncertain.
High vn_GEND_reg is masculine (it correlates +0.87 with chest resonance and −0.55 with head
resonance across the 500 per-voice means); the inverted prose in caption_general in the older
corpus release is a captioning bug and was never the source here.
The two labels agree on 89.2 % of the 379 voices this classification calls confidently — which
is also a fourth independent confirmation of the polarity, since an inverted ladder would give
~11 % — and they disagree on 41. names.csv flags every one (agrees_with_card = NO).
The 41 track speaker-identity failure. Disagreement rate falls monotonically with how well a
voice reproduces its own reference: 15.8 % in the weakest quartile by frac_ge06 (share of
takes with spk_sim >= 0.60) down to 3.2 % in the strongest. frac_ge06 differs significantly
between agreeing and disagreeing voices (0.248 vs 0.181, Welch p = 0.011), as does
min_spk_effective (p = 0.032). By family: char 21.1 %, emolia 10.4 %, mediathek 7.0 %,
anime and refvoice 0 %.
It does not explain every case, though: the bare weak/strong split is not significant on its own
(chi-square p = 0.10), and seven of the ten largest disagreements are char voices specced
Androgynous that measure 5.1–5.3 — vague spec rather than failed generation. §12's caveats
apply; nobody has listened to any of them.
The uncertain bucket is mostly measured, not designed, ambiguity. Of the 121, only 12 are
specced Androgynous; 69 are specced Female and 40 Male but do not render consistently on one
side. Within-voice std(vn_GEND_reg) averages 1.288 there against 0.759 for decided
voices — they genuinely swing take to take. Filter card_gender == "Androgynous" if you want only
the designed-androgynous voices.
Reference audio is here, not in moss-voice-profile-references
Measured 2026-08-23 from that repo's own metadata.parquet, because earlier versions of this card
described it as the reference set for these voices and that is wrong:
field
value
distinct voices
1 — k325_age3_bg1
distinct profile
4 (PEX3, PRGN4, RGN3, RW)
distinct variant
4 (raw, raw_sidon, vc, vc_sidon)
rows
106,424
overlap with the 500 profiles here
1 of 500
It is a single-voice reference / voice-conversion ablation study. Useful for what it is, but it does
not contain the reference clips of these 500 voices and never did.
The 500 reference clips are the voices/<voice>/reference.wav files in this repo, one per
profile, with reference_file / reference_variant / reference_best_version in
manifest.parquet. There is no standalone published dataset of them, so this release is the
reference-audio artefact.
10. manifest.parquet — the 56 columns
One row per voice, 500 rows, 56 columns. manifest.json carries identical rows, and each voice's own row is duplicated standalone at voices/<voice>/voice.json.
Identity and recipe
column
type
meaning
voice
string
Voice identifier. The primary key, and the directory name under voices/.
src_tree
string
Which build produced the shipped weights: repair (the 500-voice identity-repair run, 380 rows), lora3k (the 3,000-clip retrain, 115), lora3k_fc (the originals-only floor-capped retrain, 4), pvlorarelease (the single-voice reference study, emolia_c1699 only, 1).
recipe
string
repair_single_stage = one stage, 6 epochs, pool by the 0.60-or-relaxed rule. retrain_top3000_floor040 = one stage, 6 epochs, pool = top 3,000 takes by speaker similarity with a hard 0.40 floor. retrain_floorcap_originals_only = the same but originals only, every original take above the floor and no more. reference_study_2stage = the two-stage curriculum used only for emolia_c1699.
stage
string
stage1 for every row. These are first-stage identity adapters.
rank
int
LoRA rank. 4 for every voice. (Access as m["rank"] — it shadows a DataFrame method.)
lora_alpha
int
LoRA alpha. Always 2 × rank, i.e. 8.
n_target_modules
int
Number of names in target_modules. 23 for every voice.
base_model
string
The base checkpoint the adapter attaches to. Identical for all 500.
Training pool
column
type
meaning
pool_rows
int
Number of training clips the shipped adapter was fitted on.
pool_rows_stage2_sharpen
float
emolia_c1699 only: size of the second-stage sharpening pass on top of pool_rows. Null for every other voice.
pool_mean_spk
float
Mean ECAPA cosine to the voice's reference over the training pool. Higher = a purer pool. Null for emolia_c1699.
min_spk_effective
float
The lowest speaker similarity that actually entered the pool — the threshold the pool realises, whatever rule produced it. This is the number to read, not the nominal threshold.
pool_max_spk
float
Highest speaker similarity in the pool. Only recorded for retrained voices.
pool_n_orig
int
How many pool clips are ORIGINAL generator output.
pool_n_repaired
int
How many pool clips are REGENERATED by the identity-repair pipeline. 0 on every shipped row.
used_repaired
bool
True when the pool had to be topped up with regenerated audio. False on every shipped row.
Training and evaluation
column
type
meaning
epoch_chosen
string
The epoch whose checkpoint is shipped, chosen as the minimum held-out loss over the epochs trained. stage2 for emolia_c1699.
epochs_run
int
How many epochs were trained before the best was picked. 6 for 499 voices; 1 for emolia_c1699.
val_loss
float
Held-out validation loss of the shipped adapter. Lower is better. Comparable across voices only together with base_val_loss, since each voice has its own held-out set.
base_val_loss
float
Held-out loss of the frozen base model on the same rows, adapter disabled.
gain
float
base_val_loss − val_loss. The headline number. Always positive here — an adapter that did not beat the base was never shipped.
val_rows
float
Number of held-out clips the loss was measured on, where recorded — populated for 1 row only; null elsewhere.
val_gids
float
Number of held-out groups, where recorded. Null on all rows in this build.
gpu_h
float
GPU-hours of the training stage that produced the shipped weights. Encoding not included, discarded challengers not included.
Retrain bookkeeping
column
type
meaning
retrained
bool
True if the shipped weights come from this release's retrain pass (119 rows).
retrain_attempted
bool
True if this voice was one of the 123 retrain targets, whether or not the retrain won.
retrain_improved
bool
True if the retrained adapter beat the incumbent on held-out loss and was shipped.
Which candidate won: incumbent, retrain_3k or floorcap. Null for voices never targeted.
candidates_tried
int
How many adapters were trained and compared (2 = incumbent + 3,000-clip retrain; 3 = both plus the originals-only floor-capped variant, the six sub-floor voices). Null for non-targets.
old_val_loss
float
The old_* columns always describe the adapter that was not shipped, measured on the same held-out rows. For shipped_retrain that is the replaced incumbent; for rejected_worse it is the retrain that lost. Null where only one adapter ever existed.
old_gain
float
Gain over base of the adapter that was not shipped.
old_pool_rows
int
Pool size of the adapter that was not shipped.
delta_val_loss
float
loser_val_loss − winner_val_loss — how much the shipped adapter wins by. Always > 0.
Quality flags
column
type
meaning
below_saturation
bool
True if pool_rows < 3000: the adapter is on the steep part of the pool-size/quality curve and is expected to be weaker. 8 rows.
sub_floor_voice
bool
True for the six voices whose ORIGINAL takes could not supply 3,000 clips above the 0.40 identity floor. 6 rows.
Reference clip
column
type
meaning
reference_file
string
Path, relative to the repo root, of this voice's reference clip. The base model is reference-conditioned: you need this file as well as the weights.
reference_variant
string
Which enhancement variant is shipped (orig, sidon or cbx), taken from the profile card's best_version.
reference_best_version
string
The variant the profile card names as best, by DNSMOS. Normally equal to reference_variant.
Profile card
column
type
meaning
voice_name
string
Human-readable name, e.g. "Grave Documentary Narrator".
tagline
string
One-line description of the voice.
card_gender
string
Intended gender, from the profile card — the field to use. Not derived from the buggy caption_general prose, and not a measurement of the produced audio.
card_age
string
Intended age range.
card_language
string
Primary language of the card. The corpus itself is English and German. Null for 115 voices.
Comma-joined descriptive tags. Useful for filtering.
Paths and integrity
column
type
meaning
adapter_dir
string
This voice's adapter directory relative to the repo root (voices/<voice>). On the Hub, use this one.
adapter_dir_abs
string
Absolute path on the build machine. Provenance only.
adapter_weights_abs
string
Absolute path to adapter_model.safetensors on the build machine. Provenance only.
adapter_config_abs
string
Absolute path to adapter_config.json on the build machine. Provenance only.
release_root
string
Absolute path of the release root as built. Provenance only.
adapter_src
string
The build tree the weights were copied FROM. Provenance only — never load from this path.
adapter_sha256
string
SHA-256 of the shipped adapter_model.safetensors. All 500 distinct.
adapter_bytes
int
Size of the shipped adapter_model.safetensors in bytes. Sums to 17,215,216,000.
Useful slices:
python
1import pandas as pd
2m = pd.read_parquet("manifest.parquet")34m.nlargest(10,"gain")[["voice","voice_name","pool_rows","gain"]]# the strongest5m[m["below_saturation"]| m["sub_floor_voice"]]# everything flagged weak6m[m["retrain_attempted"]].groupby("retrain_status")["gain"].describe()# what the retrain did
The base model. Every adapter here attaches to it: a 4.55 B reference-conditioned TTS model — roughly a 4 B semantic transformer plus a ~550 M talker head. ~9.1 GB in bf16.
The audio codec. 12 codebooks × 1024 entries, 12.5 fps exactly (hop 3,840 at 48 kHz). Pass it as codec_path= to the processor; nothing decodes without it.
Despite the name, one voice, not 500 — a reference/voice-conversion ablation on k325_age3_bg1 alone (106,424 rows, 4 pipeline variants). It is not the reference set for this release; see the note in §9.
The ECAPA speaker embedder behind every spk_sim number in this release.
A further consolidated reference-voice dataset exists internally (TTS-AGI/moss-reference-voices-consolidated) but is private and not linkable — it is named only so the provenance chain is complete.
12. Limitations in full
They are not validated perceptually. Every quality number in this release is a held-out language-modelling loss on MOSS codes, plus ECAPA speaker cosine. No listening test was run. "Higher gain" means "the model predicts this voice's audio tokens better", which is a proxy for, not a measurement of, how much it sounds like the voice.
gain is not strictly comparable across voices. Each voice has its own held-out set, so absolute losses live on different scales. The valid comparison is adapter vs. base within a voice — which is exactly what gain is.
8 voices are below the saturation point and 6 are sub-floor. They ship flagged and named (§8), not silently.
They are not speaker clones of real people. Every profile is synthetic. There is no real identity to recover from these weights. Please do not present output as a real person's voice.
They are not a substitute for the reference clip. Generation without a matching reference gives a random speaker with the adapter's colouring.
The 0.40 and 0.60 thresholds do not transfer to another speaker embedder (§6).
Identity is bought with expressiveness. Hard filtering on speaker similarity selects for clips that sound like the profile, not for clips that are expressive or well articulated; the project's reference run measured −11 % target emotion strength and −14 % blend for its identity gain.
They are not stackable. Nothing here was measured with two voice adapters attached at once, or at a merge scale other than 1.0.
Languages are English and German only. The corpus contains nothing else.
They inherit the base model's failure modes, including empty decodes — always check audio_codes_list before using the output.
Gender and background-noise prompting is unreliable. See below.
The caption_general polarity bug
Two of the descriptive axes in the corpus's rendered caption_general prose — GEND (gender) and BKGN (background noise) — are rendered with inverted polarity. It is a bug in the prose only: the numeric vn_GEND_* / vn_BKGN_* columns are correct and authoritative.
Why it affects these adapters, and how much. The training instruction for each clip is regenerated every epoch from the numeric VoiceNet buckets, and that path is unaffected. But with probability 0.35 the generator instead uses the clip's authored caption_generalverbatim — the buggy prose. So roughly a third of the instructions each adapter saw during training carried inverted gender and background-noise wording. This applies to all 500 adapters equally, retrained and untouched alike. It is a property of the corpus these adapters were built from, and it cannot be corrected by retraining alone.
Practical consequence: do not rely on gender or background-noise wording in a prompt to steer these adapters on those two axes — the association they learned there is unreliable. All other axes are unaffected. And do not read a voice's gender out of caption_general; use card_gender (§9). This README deliberately never describes a voice using caption_general.
How the polarity was established, and why the obvious test cannot work
caption_general is generated deterministically from the numeric buckets, so it agrees with them by construction whichever way the ladder points. Comparing the prose against the bucket therefore proves nothing — a keyword check of exactly that kind over 1,112 gender-mentioning captions across 47 voices returned 52 % agreement, i.e. chance. The polarity has to be pinned against acoustic correlates instead:
Chest-versus-head resonance is the physiologically correct masculine/feminine marker, and all four correlates agree: the bucket the prose calls "feminine" is acoustically the masculine end. For BKGN the same approach gives corr(vn_BKGN_reg, vn_RCQL_reg) = +0.786 and corr(vn_BKGN_reg, qual_background_quality) = +0.402, the latter from an independent model head. Caveat: this is an acoustic-correlate argument. No listening test was run.
Provenance note: the corpus's published MOSS codes are unusable
The adapters here were trained from MOSS codes re-encoded directly from the source mp3s. They do not use the *.moss.npy arrays published in the vprof_base / vprof_repaired WebDataset copies of the corpus, and neither should you.
Those published codes encode half-speed audio. The corpus tokeniser decoded mp3s through a path that did frame.to_ndarray().reshape(-1), which is correct only for mono; PyAV returns (channels, nb_samples) for planar layouts and (1, nb_samples × channels) for packed ones, and a blind flatten turns both into a signal of length nb_samples × channels. The voice-profile audio is duplicated mono written as a 2-channel mp3 (max |L−R| = 0.000000), so the flatten produced [s0,s0,s1,s1,…] — a clean half-speed signal of exactly twice the length. Measured and independently confirmed twice: true duration 15.92 s against a stored dur_s of 31.84 s (ratio exactly 2.000), and moss_frames / true_duration = 25.00 fps against the nominal 12.5. vprof_base (1,907 shards) and vprof_repaired (25,993 shards) are affected; the mono datasets are not. The mp3s themselves are fine — only the derived codes and durations are wrong, and re-encoding from the audio, which is what this release did, recovers everything.
The methodological lesson is worth more than the bug.dur_s and moss_frames were both doubled, so the obvious integrity check — moss_frames == floor(dur_s × 12.5) — passed on every affected row, and a first scope test built on that check reported "0 affected shards". An internal-consistency check cannot detect a shared-mode error: when every derived quantity is wrong in the same way, they still agree with each other. Catching it required decoding the audio and comparing against an independent measurement.
13. Reproducing any of this
Per-voice voice.json records the exact pool size, effective threshold, epoch and both loss numbers for that adapter; manifest.parquet is the same data for all 500. The build code lives in vprof/lora3k/code/ (l3codes.py pool selection + encoding, l3train.py training and epoch choice, l3pack.py selection and packaging) and the trainer it shells out to is vprof/idloop/code/idlora.py, unchanged.
The two figures in this README that are not recomputable from the shipped manifest are the total GPU-hours including discarded challengers (≈406 GPU-h; the manifest's gpu_h column sums to 378.7 over shipped stages only) and the per-epoch validation curves behind the "472/499 strictly monotone" statement, which live in the build logs rather than the release. Everything else in §4 through §8 was recomputed from manifest.parquet and the shipped adapter_model.safetensors files when this README was written.
Citation
bibtex
1@misc{laion_moss_voice_profile_loras_500,
2 title = {MOSS Voice-Profile LoRAs: 500 synthetic character voices for
3 moss-tts-local-transformer-4.55b-voice-acting-v2},
4 author = {{LAION}},
5 year = {2026},
6 howpublished = {\url{https://huggingface.co/laion/moss-voice-profile-loras-500}}
7}
You are free to share and adapt this material for any purpose, including commercially, provided you give appropriate credit, link to the licence, and indicate if changes were made.
Two things the licence does not cover, and which you must check separately:
Synthetic voices are still voices. Every profile here is invented and no adapter is trained on a real speaker, but generated speech can still be used to deceive. Do not present output from these adapters as a recording of a real person, and disclose synthetic speech where a listener could reasonably mistake it for human.
