XCodec Mini - Neural Audio Codec

Model Description

Key Features

• Dual Encoding Architecture
  • Semantic encoder for high-level musical features
  • Acoustic encoder for detailed sound information
  • Multi-scale processing for efficient compression
• Advanced Compression
  • Multiple codebooks for flexible quality/size tradeoff
  • Support for 44.1kHz high-fidelity audio
  • Separate processing paths for vocals and instrumentals
• Technical Specifications
  • Input: Raw audio at 44.1kHz
  • Output: Compressed representations and reconstructed audio
  • Model Size: [Add total size]
  • Compression Ratio: [Add typical ratio]

Intended Uses

• High-quality music compression
• Audio archival and storage
• Music streaming applications
• Audio processing pipelines

Training Data

• Various genres and styles
• Vocal and instrumental tracks
• High-quality studio recordings

Performance and Limitations

Strengths

• High-quality audio reconstruction
• Efficient compression ratios
• Separate handling of vocals and instrumentals
• Support for high sample rates

Limitations

• Computationally intensive for real-time applications
• Requires significant GPU memory
• Best suited for offline processing
• May introduce artifacts in extreme compression settings

Technical Specifications

Model Architecture

1. Semantic Encoder
   • Based on HuBERT architecture
   • Captures high-level musical features
   • Outputs semantic tokens
2. Acoustic Encoder
   • Multi-scale convolutional architecture
   • Processes detailed sound information
   • Generates acoustic tokens
3. Dual Decoders
   • Separate decoders for vocals and instrumentals
   • Multi-stage reconstruction process
   • Quality-focused design

Input Requirements

• Audio Format: WAV/MP3
• Sample Rate: 44.1kHz
• Channels: Mono/Stereo
• Bit Depth: 16-bit

Output Format

• Reconstructed Audio: 44.1kHz WAV
• Intermediate Representations: Compressed tokens

Usage Guidelines

Hardware Requirements

• GPU: NVIDIA GPU with 8GB+ VRAM
• RAM: 16GB+ recommended
• Storage: SSD recommended for faster processing

Software Requirements

• Python 3.8+
• PyTorch 2.0+
• CUDA 11.0+
• Additional dependencies listed in installation guide

Ethical Considerations

• Copyright: Users should ensure they have proper rights to process copyrighted material
• Attribution: Proper attribution should be given when using this model
• Data Privacy: Consider data privacy implications when processing sensitive audio

Additional Information

Model Weights

• Semantic Encoder
• Vocal Decoder
• Instrumental Decoder
• Final Checkpoint

Contact