You can deploy this model and start earning money today!

TTS-AGI/
800m-adaln-test

Name: 800m-adaln-test
Availability: InStock

Views

No views yet

Echo 800M - AdaLN-Zero Fine-Tuned

Fine-tuned Echo 800M DiT TTS model with AdaLN-Zero conditioning on 59 speech attribute scores.

What is AdaLN-Zero?

Adaptive Layer Normalization (AdaLN) is a technique for injecting conditioning signals into a neural network by dynamically computing the scale and shift parameters of Layer Normalization. Instead of using expensive cross-attention (O(N²)) for style conditioning, AdaLN modulates every layer through simple affine transformations computed from the conditioning vector.

"Zero" refers to the initialization strategy: the new conditioning projection layers are initialized with zero weights. This means:

At the start of training, the score conditioning contributes nothing - the model behaves identically to the pretrained model
The model gradually learns to incorporate the scores during fine-tuning
This prevents catastrophic forgetting and ensures stable training

Architecture

Score Vector (59-dim)
    → score_dense_0 (59 → 256, SiLU activation)
    → score_dense_1 (256 → 256, ZERO-initialized)
    → Added to Timestep Embedding (256-dim)
    → cond_dense_0 (256 → 1280)
    → SiLU
    → cond_dense_1 (1280 → 1280)
    → Per-block AdaLN: scale/shift/gate for all 24 Transformer blocks

59 Conditioning Dimensions

Emotion Categories (40 dimensions)

#	Dimension	Description
0	Affection	Warmth, love, caring tone
2	Amusement	Finding something funny
3	Anger	Frustration, rage, hostility
5	Astonishment_Surprise	Shock, being caught off guard
7	Awe	Wonder, reverence
9	Bitterness	Resentment, cynicism
10	Concentration	Focused, deliberate speech
12	Confusion	Uncertainty, bewilderment
13	Contemplation	Thoughtful, reflective
14	Contempt	Disdain, looking down
15	Contentment	Satisfaction, peace
16	Disappointment	Letdown
17	Disgust	Revulsion
18	Distress	Suffering, anguish
19	Doubt	Skepticism
20	Elation	Joy, happiness
21	Embarrassment	Self-consciousness
22	Emotional_Numbness	Flatness
23	Fatigue_Exhaustion	Tiredness
24	Fear	Anxiety, being scared
26	Helplessness	Feeling powerless
28	Hope_Enthusiasm_Optimism	Positive outlook
29	Impatience_and_Irritability	Annoyance
30	Infatuation	Intense attraction
31	Interest	Engagement, curiosity
32	Intoxication_Altered_States	Altered speech
33	Jealousy_&_Envy	Coveting
34	Longing	Yearning
35	Malevolence_Malice	Ill intent
37	Pain	Physical/emotional pain
38	Pleasure_Ecstasy	Delight, bliss
39	Pride	Accomplishment
41	Relief	Release of tension
42	Sadness	Sorrow, melancholy
44	Sexual_Lust	Desire
45	Shame	Guilt
47	Sourness	Acidity in tone
49	Teasing	Playful mockery
50	Thankfulness_Gratitude	Appreciation
51	Triumph	Victory

Vocal Attributes (15 dimensions)

#	Dimension	Description
1	Age	Speaker age (higher = older)
4	Arousal	Energy level (low=calm, high=excited)
6	Authenticity	How genuine the speech sounds
8	Background_Noise	Amount of background noise
11	Confident_vs._Hesitant	Confidence level
25	Gender	Vocal gender (neg=male, pos=female)
27	High-Pitched_vs._Low-Pitched	Vocal pitch
36	Monotone_vs._Expressive	Expressiveness
40	Recording_Quality	Recording quality
43	Serious_vs._Humorous	Tone (low=serious, high=funny)
46	Soft_vs._Harsh	Voice texture
48	Submissive_vs._Dominant	Dominance
52	Valence	Positive/negative (high=positive)
53	Vulnerable_vs._Emotionally_Detached	Vulnerability
54	Warm_vs._Cold	Voice warmth

Audio Quality Scores (4 dimensions)

#	Dimension	Description	Range
55	score_background_quality	Background cleanliness	~1-5 MOS
56	score_content_enjoyment	Content engagement	~1-5 MOS
57	score_overall_quality	Overall audio quality	~1-5 MOS
58	score_speech_quality	Speech clarity	~1-5 MOS

Usage

Installation

pip install numpy

Quick Start

python

from inference import create_score_vector, create_preset_scores, load_checkpoint, generate_latents
import numpy as np

# Load the fine-tuned model
params = load_checkpoint("checkpoints/step_0019500.pkl")

# Create conditioning scores
# Option 1: Set specific dimensions
scores = create_score_vector(
    Amusement=2.5,      # Amused/funny tone
    Interest=3.0,        # Engaged, curious
    Valence=2.0,         # Positive mood
    score_overall_quality=3.5,  # High quality
)

# Option 2: Use a preset
scores = create_preset_scores('happy')  # or 'angry', 'sad', 'calm', 'excited', etc.

# Prepare speaker reference (from your audio codec latents)
ref_latent = np.load("your_speaker_ref.npy")  # shape: (frames, 128)

# Generate speech latents
latents = generate_latents(
    params,
    text="Hello! This is emotion-conditioned speech.",
    ref_latent=ref_latent,
    scores=scores,
    num_frames=256,   # ~10 seconds at 25fps
    num_steps=50,     # ODE solver steps
)
# latents shape: (256, 128) - decode with your audio codec

Conditioning on Single vs Multiple Dimensions