LaionBox v0.2-wip — Fine-tuned DramaBox with Multi-Auxiliary Losses

Key Improvements over Vanilla DramaBox

• Higher naturalness: +11% CLAP naturalness score
• Better voice cloning: +2% speaker similarity (WavLM-SV cosine)
• Reduced artifacts: Quality probability 0.99+ (vs ~0.85 baseline)
• Less background noise: Trained on enhanced, high-quality reference audio
• More expressive: Trained on diverse voice acting scenarios, emotions, and archetypes

Model Details

Auxiliary Losses

1. CLAP Naturalness (ratio=3.0): Maximizes cosine similarity between generated audio's CLAP embedding and positive text prompts ("realistic, genuine, spontaneous, authentic, natural") while minimizing similarity to negative prompts ("distorted, unnatural, robotic")
2. Quality MLP (included with naturalness): Binary classifier (768→256→64→1) trained on VoiceCLAP embeddings to distinguish real from synthetic audio
3. Centroid Real/Fake (ratio=3.0): Pushes audio embeddings toward the centroid of real speech embeddings and away from synthetic speech centroid
4. Speaker Similarity (ratio=3.0): WavLM-SV cosine similarity between reference and generated speaker embeddings
5. Comb Filter Detector (ratio=3.0): CNN operating on latent space to detect comb-filter artifacts
6. Artifact Detector V2 (ratio=3.0): Larger residual CNN for general artifact detection in latent space

Critical Innovation: Differentiable Reward Chain

Files

.
├── lora_weights.safetensors          # Main LoRA checkpoint (865 MB)
├── training_config.yaml              # Training configuration
├── training_args.yaml                # Full training arguments
├── discriminators/
│   ├── quality_classifier.pt         # Quality/MOS prediction MLP
│   ├── real_fake_classifier.pt       # Real vs AI detection MLP
│   ├── best_artifact_detector_v2.pt  # CNN artifact detector (latent space)
│   ├── best_comb_detector.pt         # CNN comb filter detector (latent space)
│   └── best_clap_medium.pt          # CLAP-based artifact MLP
├── scripts/
│   ├── dramabox_finetune_train_multi_aux.py  # Main training script (6-aux)
│   ├── dramabox_finetune_train.py            # Base training script
│   ├── dramabox_finetune_prepare.py          # Data preparation
│   ├── train_artifact_detector_v2.py         # Artifact detector training
│   ├── train_artifact_detector_clap.py       # CLAP artifact detector training
│   ├── train_comb_filter_detector.py         # Comb filter detector training
│   └── train_binary_classifiers.py           # Quality/real-fake classifier training
└── configs/
    └── *.yaml                                # All training configurations

Discriminator Models

Quality Classifier (discriminators/quality_classifier.pt)

• Architecture: MLP (768→128→32→1, sigmoid)
• Input: VoiceCLAP-small embeddings (768-dim)
• Training: Binary classification (real=1, synthetic=0)
• Accuracy: 100% on validation set
• Purpose: Provides quality probability signal during training

Real/Fake Classifier (discriminators/real_fake_classifier.pt)

• Architecture: Same as quality classifier
• Training: Distinguishes real recordings from TTS outputs
• Purpose: Centroid-based distribution matching

Artifact Detector V2 (discriminators/best_artifact_detector_v2.pt)

• Architecture: Residual CNN (8→64→128→256→384→512, ~10.6M params)
• Input: Latent tensors [B, 8, T, 16]
• Training: Binary classification of clean vs artifacted latents
• Purpose: Detects vocoder artifacts directly in latent space (no decoding needed)

Comb Filter Detector (discriminators/best_comb_detector.pt)

• Architecture: CNN (8→32→64→128→128, ~1M params)
• Input: Latent tensors [B, 8, T, 16]
• Training: Detects comb-filter interference patterns
• Purpose: Penalizes latent configurations that produce comb artifacts

CLAP Artifact MLP (discriminators/best_clap_medium.pt)

• Architecture: MLP (768→512→256→64→1, LayerNorm, GELU)
• Input: VoiceCLAP embeddings (768-dim)
• Training: Distinguishes clean from artifacted audio in embedding space
• Purpose: Perceptual artifact detection via CLAP representations

Usage

Known Issues

• Subtle metallic/ringing artifacts: BigVGAN v2 vocoder produces harmonic artifacts when processing LoRA-modified mel spectrograms that are slightly out-of-distribution
• Not from stereo interference: L-R channel correlation is >0.999; artifacts are per-channel, intrinsic to the vocoder
• Root cause: Snake/SnakeBeta activations amplify small spectral deviations into audible harmonics

Training Data Sources

1. DramaBox Voice Acting (~3,247 samples): Procedurally generated prompts from EmoNet/voice acting taxonomies, expanded by Gemma-4 LLM, covering diverse emotional scenarios and character archetypes
2. Podcast Data (~11,966 samples): High-quality conversational speech from TTS-AGI/podcast-tokenized, decoded via DAC-VAE and annotated with Whisper

License

Citation