Speech Recognition AI: Fine-Tuned Whisper and Wav2Vec2 for Real-Time Audio
This project fine-tunes OpenAI's Whisper (whisper-small) and Facebook's Wav2Vec2 (wav2vec2-base-960h) models for real-time speech recognition using live audio recordings. It’s designed for dynamic environments where low-latency transcription is key, such as live conversations or streaming audio.
Model Description
Fine-tuned Whisper and Wav2Vec2 models for real-time speech recognition on live audio.
Features
Real-time audio recording: Captures live 16kHz mono audio via microphone input.
Continuous fine-tuning: Updates model weights incrementally during live sessions.
Speech-to-text transcription: Converts audio to text with high accuracy.
Model saving/loading: Automatically saves fine-tuned models with timestamps.
Dual model support: Choose between Whisper and Wav2Vec2 architectures.
Usage
Start Fine-Tuning
Fine-tune the model on live audio:
bash
1# For Whisper model2python main.py --model_type whisper
34# For Wav2Vec2 model5python main.py --model_type wav2vec2
Records audio in real-time and updates the model continuously. Press Ctrl+C to stop training and save the model automatically.
Transcription
Test the fine-tuned model:
bash
1# For Whisper model2python test_transcription.py --model_type whisper
34# For Wav2Vec2 model5python test_transcription.py --model_type wav2vec2
Records 5 seconds of audio (configurable in code) and generates a transcription.
speech-model/
├── dataset.py # Audio recording and preprocessing
├── train.py # Training pipeline
├── test_transcription.py # Transcription testing
├── main.py # Main script for fine-tuning
├── README.md # This file
└── requirements.txt # Dependencies
Training Data
The models are fine-tuned on live audio recordings collected during runtime. No pre-existing dataset is required—users generate their own data via microphone input.
Evaluation Results
Future updates will include WER (Word Error Rate) metrics compared to base models.
