It has been trained using a Frozen Feature Extractor strategy to preserve the model's acoustic understanding while adapting to emotion detection. This approach ensures stable performance and prevents "Catastrophic Forgetting," achieving nearly 80% accuracy on the validation set.

Update: The "Calm" and "Neutral" classes have been merged to improve classification consistency, resulting in 7 distinct emotion classes.

Model Description

Model Architecture: Wav2Vec2 with a frozen CNN feature extractor and a trainable sequence classification head.
Language: English
Task: Speech Emotion Recognition (SER)
Fine-tuned from: facebook/wav2vec2-base-960h
Class Merging: The "Calm" emotion has been merged into "Neutral" to reduce label ambiguity.

Datasets

The model was trained on a combined dataset of ~12,000 audio files from:

Performance Metrics

Metric	Score
Accuracy	79.94%
F1 Score	79.65%
Validation Loss	0.644

Note: Achieved stable convergence using a Cosine Learning Rate Scheduler and Frozen Feature Extractor.

Supported Emotions (7 Classes)

The model classifies audio into one of the following emotions:

😠 Angry
🤢 Disgust
😨 Fear
😊 Happy
😐 Neutral (includes Calm)
😢 Sad
😲 Surprise

Training Configuration

The model was fine-tuned using the following "Robust" configuration to ensure stability on Google Colab T4 GPUs:

Feature Extractor: FROZEN (Locked weights to preserve pre-trained acoustic features)
Epochs: 10
Learning Rate: 3e-5
Scheduler: Cosine Decay (Smooth landing)
Batch Size: 4 (Physical) / 16 (Effective via Gradient Accumulation)
Optimizer: AdamW
Precision: fp32 (Standard precision to prevent gradient underflow)

Limitations

Audio Requirements:

Sampling Rate: 16kHz (Model will hallucinate if input is not 16kHz. Resampling is required.)
Duration: < 10 seconds recommended (Longer files should be chunked).
Environment: Best results with clear speech; background noise may reduce confidence.

Demo

Try the model in your browser: 👉 Hugging Face Space: Audio Emotion Recognition

FINETUNING NOTEBOOOKS

Notebook

Contact

GitHub: DGautam11
LinkedIn: Deepan Gautam
Hugging Face: @Dpngtm

For issues and questions, feel free to:

Open an issue on the Model Repository
Comment on the Demo Space

Usage

python

from transformers import AutoModelForAudioClassification, Wav2Vec2Processor
import torch
import torchaudio
import numpy as np

# 1. Load Model and Processor
model_id = "Dpngtm/wav2vec2-emotion-recognition"
model = AutoModelForAudioClassification.from_pretrained(model_id)
processor = Wav2Vec2Processor.from_pretrained(model_id)

# 2. Load Audio
# Replace with your file path
audio_path = "path_to_audio.wav" 
speech_array, sampling_rate = torchaudio.load(audio_path)

# 3. Resample to 16kHz (CRITICAL STEP)
if sampling_rate != 16000:
    resampler = torchaudio.transforms.Resample(sampling_rate, 16000)
    speech_array = resampler(speech_array)
    sampling_rate = 16000

# 4. Handle Stereo (Convert to Mono)
if speech_array.shape[0] > 1:
    speech_array = torch.mean(speech_array, dim=0, keepdim=True)

# 5. Process & Predict
inputs = processor(speech_array.squeeze(), sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(**inputs).logits

# 6. Decode Result
predicted_id = torch.argmax(logits, dim=-1).item()
id2label = model.config.id2label
predicted_label = id2label[predicted_id]

print(f"Predicted Emotion: {predicted_label}")