We used Python 3.9.16, PyTorch 2.6.0, and PyTorch-Lightning 2.5.0 to train and test our models.
Portions of this code are adapted from OpenAI's Whisper.
To set up the project environment using conda, follow these steps:
💡 Make sure you have Miniconda or Anaconda installed before proceeding.
Create the conda environment
conda env create -f environment.yml
Activate The environment
conda activate careless_whisper
Install the appropriate PyTorch version
Depending on your hardware and CUDA version, install PyTorch by following the instructions at https://pytorch.org/get-started/locally.
This project was tested with CUDA 12.4, but it should also work with compatible earlier or later versions.
After installing all of the dependencies, you can try to run inference.
🤖 Available Models
We fine-tuned three different sizes of Whisper, all support english only transcription.
A large-v2 that was fine tuned on multilingual data is available, and supports English, French, Spanish, German and Portuguese with chunk size of 300 miliseconds.
Size
Chunk Size [msec]
Multilingual
base
40, 100, 200, 300
N/A
small
40, 100, 200, 300, 1000
N/A
large-v2
40, 100, 200, 300, 1000
300
🎤 Running Inference
To run inference, download the repo content, and run from the repository root accroding to following sections.
Note: The models are hosted on the Hugging Face Hub, which requires an access token.
Make sure you are logged in with your token to access the models.
Click on "New token", give it a name, select the appropriate scopes (usually read is enough), and create it.
Login using the Hugging Face CLI:
Install the CLI if you don’t have it:
pip install huggingface_hub
Then login:
huggingface-cli login
Paste your token when prompted.
🖥️ CLI Usage
The transcription model is easily activated using the next command:
bash
1# Using a local microphone for streaming transcription, dumping the recording to out.wav2python transcribe.py \3--output_filename out.wav \4--channels 2\5--model small \6--chunk_size 300\7--device cuda \8--beam_size 5\9--ca_kv_cache \
A simulation of a stream on a wav file is also available:
bash
1# Simulating a stream on a wav file2python transcribe.py \3--model small \4--chunk_size 300\5--device cuda \6--beam_size 5\7--ca_kv_cache \8--wav_file /path/to/audio.wav \9--simulate_stream \10--use_latency
🐍 Python Usage
If you prefer using python, a code sinppet utilizing a microphone or a wav file is provided below:
python
1import torch
2import careless_whisper_stream
34model_size ="small"# model size5chunk_size =300# chunk size in milliseconds6multilingual =False# currently on large-v2_300msec supports other languages than english.7device ="cuda"if torch.cuda.is_available()else"cpu"89model = careless_whisper_stream.load_streaming_model(name=model_size,10 gran=chunk_size,11 multilingual=multilingual,12 device=device)1314# using a local microphone recording 15texts_microphone = model.transcribe(output_filename="/path/to/dump/file.wav",16 channels=2,17 beam_size=5,18 ca_kv_cache=True)1920# Simulating on a wav file21texts_wav_simulation = model.transcribe(simulate_stream=True,22 wav_file="/path/to/file/you/want/to/transcribe.wav",23 beam_size=5,24 ca_kv_cache=True)
🦾 Training
In order to train using LoRA, you can use our existing code. Make sure all the requirements are installed.
📂 Dataset Structure
Before starting model training using the command-line interface provided below, you must first configure your dataset dictionary file located at training_code/ds_dict.py.
This file defines a Python dictionary named ds_paths, where you should specify paths to the train, val, and test partitions of your dataset. Each partition should be a CSV file with the following three columns:
wav_path — Path to the WAV audio file.
tg_path — Path to the corresponding .TextGrid file containing forced alignment.
raw_text — Ground truth transcription.
Note: The dictionary key (i.e., the name of the dataset) will be used by the training script to identify and load the dataset correctly.
You can find an example entry in training_code/ds_dict.py.
