Model Card for amirhossein-yousefi/speech2text-intensity-regression-wav2vec

Model Details

Model Description

• Developed by: Amirhossein Yousefi
• Model type: Multitask speech models (ASR + scalar intensity regression).
  • facebook/wav2vec2-base-960h (CTC) + attention‑masked mean pooling regressor
• Language(s): English (depends on chosen dataset/splits)
• License: MIT
• Finetuned from: facebook/wav2vec2-base-960h

Model Sources

• Repository: https://github.com/amirhossein-yousefi/speech2text-intensity-regression-wav2vec
• Demo: Gradio script in app/gradio_app.py

Uses

Direct Use

• Transcribe English speech to text (ASR) and simultaneously estimate normalized intensity for the same audio clip.
• Interactive inference via CLI or Gradio.

Downstream Use

• Domain‑specific fine‑tuning for ASR while keeping the intensity head.
• Use intensity as an auxiliary signal for VAD thresholds, diarization heuristics, or UX analytics.

Out‑of‑Scope Use

• Safety‑critical applications without human review.
• Treating the intensity output as perceptual loudness or emotion/affect; it is RMS dBFS‑derived and sensitive to mic gain/environment.

Bias, Risks, and Limitations

• Dataset bias: Default training on LibriSpeech (read audiobooks) may not reflect conversational or accented speech.
• Device & environment sensitivity: Intensity depends on microphone, distance, and preprocessing.
• Domain shift: Degradation is expected on far‑field/noisy/multilingual inputs without adaptation.

Recommendations

• Calibrate or post‑normalize intensity for your capture setup.
• Report WER and regression errors by domain (mic type, SNR buckets, etc.). Keep a human in the loop for sensitive deployments.

How to Get Started with the Model

Environment

Train (Whisper backbone)

Train (Wav2Vec2‑CTC backbone)

Evaluate

Inference (CLI)

Gradio Demo

Training Details

Training Data

• Default: librispeech_asr (train.clean.100; eval on validation.clean / test.clean).
• Optional: mozilla-foundation/common_voice_13_0 via --dataset and --language.

Training Procedure

Preprocessing

• Load/resample to 16 kHz per backbone requirements.
• Compute intensity labels from raw audio; LUFS (via pyloudnorm) can be used as an alternative.

Training Hyperparameters

• Training regime: fp16 mixed precision when available; batch size and LR configured via configs/*.yaml.

Speeds, Sizes, Times

• Example single‑epoch fine‑tuned weights are linked in the repo README (training-logs/ contains logs).

Evaluation

Testing Data, Factors & Metrics

• Testing Data: LibriSpeech test.clean by default; optionally Common Voice.
• Factors: noise level, microphone/domain, utterance length.
• Metrics:
  • ASR: Word Error Rate (WER)
  • Intensity regression: MAE, MSE, and R²

Results

📊 Training Logs & Metrics

• Total FLOPs (training): 11,971,980,681,992,470,000
• Training runtime: 9,579.8516 seconds for 3 epoch
• Logging: TensorBoard-compatible logs in src/checkpoint/logs

✅ Full Metrics

🔎 Highlights

• Validation WER (↓): 12.897% (0.128966 as fraction)
• Validation Loss: 21.7842
• Fast eval throughput: 17.05 samples/s • 4.264 steps/s

WER from jiwer.wer (fraction in [0,1]; percent shown for readability).
This run uses a CTC objective for ASR and an auxiliary intensity head (multi‑task), but only ASR metrics were logged during evaluation.

Validation (Dev)

Training Summary

Summary

Model Examination

Environmental Impact

• Hardware Type: Laptop GPU
• GPU: NVIDIA GeForce RTX 3080 Ti Laptop (16 GB VRAM)

Technical Specifications

Model Architecture and Objective

• Wav2Vec2‑CTC variant: Transformer encoder with CTC head for ASR + attention‑masked mean‑pooled regressor.

Compute Infrastructure

• Hardware: Laptop with NVIDIA RTX 3080 Ti (16 GB).
• Software: Python, PyTorch, Hugging Face transformers/datasets, Gradio.

Citation

More Information

• Configs: configs/wav2vec2_base.yaml
• Deployment: Amazon SageMaker packaging/inference under sagemaker/

Model Card Contact