You can deploy this model and start earning money today!

AudenAI/
auden-encoder-voice

Name: auden-encoder-voice
Rating: 2.5 (14 reviews)

Views

No views yet

Auden-Voice

Auden-Voice is a general-purpose voice encoder trained to learn robust speaker representations.

The model is trained using multi-task learning, where jointly optimizing speaker identification, emotion, gender, and age classification objectives leads to more general and transferable voice representations.

Model Details

Model type: Voice encoder
Architecture: Zipformer
Embedding dimension: 768
Number of parameters: ~156M
Framework: PyTorch
Output: Frame-level embeddings [B, T, D]
Pooling: User-defined (e.g., mean pooling for utterance-level embeddings)

Training

Training Strategy

Multi-task learning was found to work best. The model is jointly trained on the following tasks:

Speaker identification
Emotion classification
Gender classification
Age classification

This setup encourages the encoder to learn robust and general-purpose voice representations.

Training Data

The model is trained on publicly available academic speech datasets, totaling approximately 2050 hours of audio.

Task	Dataset(s)	#Samples	Hours
Speaker Identification	VoxCeleb2	974k	2026
Paralinguistic Tasks	CREMA-D, RAVDESS, IEMOCAP, TESS	18.3k	20

Training Code

Full training scripts and configurations are available at:
https://github.com/AudenAI/Auden/tree/main/examples/voice

Intended Use

This model is intended to be used as a general-purpose voice encoder for:

Speaker identification and verification
Speaker diarization
Emotion, gender, and age classification
Audio–text and text–audio retrieval
Speech-related downstream tasks that benefit from pretrained voice embeddings

How to Use

Load the Encoder

python

from auden.auto.auto_model import AutoModel
import torch

encoder = AutoModel.from_pretrained("AudenAI/auden-encoder-voice")
encoder = encoder.to("cuda" if torch.cuda.is_available() else "cpu")

# Extract Voice Embeddings
import torch.nn.functional as F

audio_files = ["/path/to/audio1.wav", "/path/to/audio2.wav"]
embeddings_list = []

for audio_file in audio_files:
    x, x_lens = encoder.extract_feature([audio_file])
    x, x_lens = x.to(device), x_lens.to(device)

    with torch.no_grad():
        encoder_output = encoder(x, x_lens)
        frame_embeddings = encoder_output["encoder_out"]  # [B, T, D]

        # Global average pooling (example for speaker verification)
        T = frame_embeddings.size(1)
        mask = (torch.arange(T, device=device).unsqueeze(0) < x_lens.unsqueeze(1)).unsqueeze(-1).float()
        utterance_embedding = (frame_embeddings * mask).sum(dim=1) / mask.sum(dim=1)

        embeddings_list.append(utterance_embedding)

embeddings = torch.cat(embeddings_list, dim=0)  # [N, D]
embeddings = F.normalize(embeddings, p=2, dim=-1)

similarity = torch.matmul(embeddings[0], embeddings[1])
print(f"Cosine similarity: {similarity:.4f}")


# Expected Output
🎵 Audio 1:
   Frame embeddings shape: torch.Size([1, 97, 768])
   Utterance embedding shape: torch.Size([1, 768])

🎵 Audio 2: 
   Frame embeddings shape: torch.Size([1, 138, 768])
   Utterance embedding shape: torch.Size([1, 768])

Cosine similarity: 0.7234
Same speaker: YES

Performance

Task - Dataset	Metric
Speaker Identification - VoxCeleb2	Accuracy 95.25%
Speaker Verification - VoxCeleb1-O	EER 3%
Speaker Diarization - VoxConverse	DER 17%
Age Classification - CREMA-D	Accuracy 93.91%
Gender Classification - CREMA-D	Accuracy 99.72%
Gender Classification - RAVDESS	Accuracy 100%
Emotion Classification - CREMA-D	Accuracy 83.99%
Emotion Classification - RAVDESS	Accuracy 89.71%
Audio → Text Retrieval - ParaspeechCaps	R@1 63.31
Text → Audio Retrieval - ParaspeechCaps	R@1 61.69
LLM-QA Emotion - AirBench-MELD	Accuracy 27.23%
LLM-QA Emotion - AirBench-IEMOCAP	Accuracy 84.70%
LLM-QA Gender - AirBench-MELD	Accuracy 81.58%
LLM-QA Gender - AirBench-CommonVoice	Accuracy 93.15%
LLM-QA Age - AirBench-CommonVoice	Accuracy 58.27%

Limitations

The model is trained primarily on English speech data and may not generalize well to other languages.
The model is not evaluated on generative tasks such as speech synthesis or voice conversion.
Utterance-level representations depend on the pooling strategy selected by the user.

Citation

If you use this model in your research, please cite:

bibtex

@article{huo2025auden,
  title={Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding},
  author={Huo, Mingyue and Tseng, Wei-Cheng and Shao, Yiwen and Zhang, Hao and Yu, Dong},
  journal={arXiv preprint arXiv:2511.15145},
  year={2025}
}