🎙️ # 🎙️ Whisper Large v3 — Fine-tuned for Children's Speech Recognition
🧠 What is This Model?
Imagine a smart assistant that can listen to a child speak and write down exactly what they said. That is what this model does!
Most speech recognition systems are trained on adult voices — they struggle to understand children because:
Children have higher pitched voices than adults
Children sometimes mispronounce words (e.g., "elephant" → "efant")
Children speak with different rhythm and speed
Some children have speech disorders
This model is a fine-tuned version of OpenAI's Whisper Large v3 — one of the world's best speech recognition models — specially adapted to understand children's speech .
🏆 Competition Results
This model was built for the "On Top of Pasketti: Children's Speech Recognition Challenge" on DrivenData.
Metric Score Validation WER 0.1030 (10.30%) 🔥Public Leaderboard WER 0.4432 Competition Leaderboard #1 0.1914
WER = Word Error Rate — Lower is better! If a child says 10 words and the model gets 1 wrong, WER = 10%.
📁 Files in This Repository — Complete Guide
This section explains every single file in this repository, what it contains, and why it exists.
1. model.safetensors ⭐ (Most Important File!)
What it is: The "brain" of the model — contains all the learned knowledge.
What's inside:
Millions of numbers (called "weights" or "parameters") that the model learned during training
These numbers represent everything the model knows about children's speech
Total: 1.5 billion parameters (like 1.5 billion brain connections!)
How it's used:
Loaded automatically when you run from_pretrained()
Without this file — the model cannot work at all
Size: ~3GB (large because it contains so much knowledge)
Simple analogy: Think of it like a student's brain after years of studying. All the knowledge is stored here.
2. model.safetensors.index.json
What it is: A "table of contents" for the model weights.
What's inside:
1 {
2 "metadata" : { "total_size" : 3000000000 } ,
3 "weight_map" : {
4 "encoder.layers.0.weight" : "model.safetensors" ,
5 "decoder.layers.0.weight" : "model.safetensors" ,
6 ...
7 }
8 }
How it's used:
Tells the loading code exactly where each piece of the model is stored
Needed when model is split across multiple files
You never need to open this manually
Simple analogy: Like an index page in a textbook — tells you which page (file) contains which chapter (weight).
3. config.json ⭐ (Architecture Blueprint)
What it is: The blueprint that describes how the model is built.
What's inside:
1 {
2 "model_type" : "whisper" ,
3 "d_model" : 1280 ,
4 "encoder_layers" : 32 ,
5 "decoder_layers" : 32 ,
6 "encoder_attention_heads" : 20 ,
7 "num_mel_bins" : 128 ,
8 "vocab_size" : 51866 ,
9 ...
10 }
Key settings explained:
Setting Value Meaning model_typewhisper This is a Whisper model num_mel_bins128 Uses 128 frequency bands (Large v3 specific!) encoder_layers32 32 layers for processing audio decoder_layers32 32 layers for generating text vocab_size51,866 Knows 51,866 different word pieces
How it's used:
Read first when loading the model
Tells PyTorch how to build the model structure
Must match the weights in model.safetensors
Simple analogy: Like an architect's blueprint — describes the structure before the building (weights) fills it.
4. generation_config.json
What it is: Settings that control how the model generates (writes) text.
What's inside:
1 {
2 "forced_decoder_ids" : null ,
3 "suppress_tokens" : [ 1 , 2 , 7 , 8 , ... ] ,
4 "task" : "transcribe" ,
5 "language" : "en" ,
6 "max_new_tokens" : 225
7 }
Key settings explained:
Setting Meaning task: "transcribe"Convert speech to text (not translate) language: "en"English language only max_new_tokens: 225Generate at most 225 words suppress_tokensList of tokens to never generate
How it's used:
Automatically loaded during inference
Controls the text generation process
Can be overridden by passing parameters to generate()
Simple analogy: Like rules given to a writer — "write in English", "keep it under 225 words", "don't use these specific words".
5. preprocessor_config.json ⭐ (Audio Processing Settings)
What it is: Settings for converting raw audio into a format the model understands.
What's inside:
1 {
2 "feature_size" : 128 ,
3 "sampling_rate" : 16000 ,
4 "hop_length" : 160 ,
5 "n_fft" : 400 ,
6 "padding_value" : 0.0 ,
7 "return_attention_mask" : false
8 }
Key settings explained:
Setting Value Meaning feature_size128 Uses 128 mel frequency bins sampling_rate16000 Audio must be at 16kHz hop_length160 Slide window by 160 samples n_fft400 FFT window size of 400 samples
How it's used:
WhisperFeatureExtractor reads this to process audio correctlyAudio is converted to mel spectrogram (visual representation of sound)
Without correct settings — model receives wrong input!
Simple analogy: Like settings on a camera — resolution, brightness, zoom. Must be set correctly to get a good picture.
6. processor_config.json
What it is: Configuration for the complete processor (feature extractor + tokenizer combined).
What's inside:
1 {
2 "auto_map" : {
3 "AutoProcessor" : "transformers.WhisperProcessor"
4 } ,
5 "feature_extractor_type" : "WhisperFeatureExtractor" ,
6 "tokenizer_class" : "WhisperTokenizer"
7 }
How it's used:
Tells AutoProcessor which classes to use
Links feature extractor and tokenizer together
Loaded automatically with WhisperProcessor.from_pretrained()
Simple analogy: Like a connector cable — links the audio processing part with the text processing part.
7. tokenizer.json ⭐ (Language Dictionary)
What it is: The complete tokenizer — converts between text and numbers.
What's inside:
A vocabulary of 51,866 "tokens" (word pieces)
Rules for splitting words into tokens
Special tokens like <|startoftranscript|>, <|en|>, <|endoftext|>
Example of tokenization:
"children" → [1200, 303] (converted to numbers)
[1200, 303] → "children" (converted back to text)How it's used:
Model works with numbers, not words
Tokenizer converts predictions (numbers) → readable text
Also converts training labels (text) → numbers
Simple analogy: Like a codebook used by spies — converts messages to secret codes and back.
8. tokenizer_config.json
What it is: Configuration settings for the tokenizer.
What's inside:
1 {
2 "tokenizer_class" : "WhisperTokenizer" ,
3 "language" : "english" ,
4 "task" : "transcribe" ,
5 "bos_token" : "<|endoftext|>" ,
6 "eos_token" : "<|endoftext|>" ,
7 "pad_token" : "<|endoftext|>"
8 }
Key settings explained:
Setting Meaning language: "english"Transcribe in English task: "transcribe"Speech to text task bos_tokenToken that starts a sequence eos_tokenToken that ends a sequence pad_tokenToken used for padding
Simple analogy: Like grammar rules for the codebook — when to start, when to stop, what punctuation to use.
9. vocab.json ⭐ (Vocabulary List)
What it is: Complete list of all words/pieces the model knows.
What's inside:
1 {
2 "!" : 0 ,
3 "\"" : 1 ,
4 "#" : 2 ,
5 ...
6 "children" : 5234 ,
7 "speech" : 8901 ,
8 ...
9 "<|endoftext|>" : 50256
10 }
How it's used:
Maps each token to a unique number
Used by tokenizer to encode/decode text
Contains 51,866 entries
Simple analogy: Like a dictionary — each word has a unique page number (ID).
10. merges.txt
What it is: Rules for how to combine smaller pieces into larger words.
What's inside:
#version: 0.2
Ġ t
Ġ a
h e
i n
r e
...How it's used:
Byte-Pair Encoding (BPE) tokenization
Tells tokenizer how to merge characters into subwords
Example: "ch" + "ild" + "ren" → "children"
Simple analogy: Like rules for combining LEGO pieces — small pieces combine into larger meaningful shapes.
11. added_tokens.json
What it is: Special tokens added specifically for Whisper's speech recognition tasks.
What's inside:
1 {
2 "<|endoftext|>" : 50256 ,
3 "<|startoftranscript|>" : 50258 ,
4 "<|en|>" : 50259 ,
5 "<|transcribe|>" : 50359 ,
6 "<|notimestamps|>" : 50363 ,
7 ...
8 }
Key tokens explained:
Token Meaning `< startoftranscript `< en `< transcribe `< endoftext `< notimestamps
Simple analogy: Like stage directions in a play — "BEGIN SCENE", "SPEAK IN ENGLISH", "END SCENE".
12. normalizer.json
What it is: Rules for cleaning and standardizing text output.
What's inside:
Rules to lowercase text
Rules to remove punctuation
Rules to convert numbers to words ("2" → "two")
Rules to fix common spelling variations
How it's used:
Applied to both predictions and ground truth before calculating WER
Ensures fair comparison — "Hello!" and "hello" are treated the same
This is Whisper's official English Text Normalizer
Simple analogy: Like a proofreader who standardizes all text before grading — removes extra spaces, fixes capitalization.
13. special_tokens_map.json
What it is: Maps special token names to their actual token strings.
What's inside:
1 {
2 "bos_token" : "<|endoftext|>" ,
3 "eos_token" : "<|endoftext|>" ,
4 "pad_token" : "<|endoftext|>" ,
5 "additional_special_tokens" : [
6 "<|startoftranscript|>" ,
7 "<|en|>" ,
8 "<|transcribe|>" ,
9 ...
10 ]
11 }
How it's used:
Tells the tokenizer which tokens have special meaning
Used during both training and inference
Ensures model knows when to start/stop generating
Simple analogy: Like a legend on a map — explains what each special symbol means.
🚀 How to Use This Model — Complete Step by Step Guide
Prerequisites (Things You Need First)
Step 1 — Install Python:
Download Python 3.11 from python.org
Make sure to check "Add Python to PATH" during installation
Step 2 — Install Required Libraries:
Open terminal/command prompt and run:
pip install transformers torch torchaudio soundfile librosaThis installs:
Library Purpose transformersLoads and runs the Whisper model torchDeep learning framework torchaudioAudio processing soundfileRead audio files librosaAudio analysis
Option 1 — Quickest Way (3 Lines of Code!)
1 from transformers import pipeline
2
3 # Load model directly from HuggingFace
4 transcriber = pipeline (
5 "automatic-speech-recognition" ,
6 model = "harphool17/whisper-large-v3-children-asr"
7 )
8
9 # Transcribe audio file
10 result = transcriber ( "your_audio_file.wav" )
11 print ( result [ "text" ] )
That's it! 🎉
Option 2 — Complete Control (Recommended)
1 import torch
2 import soundfile as sf
3 import librosa
4 import numpy as np
5 from transformers import WhisperProcessor , WhisperForConditionalGeneration
6
7 # ── Step 1: Load Model ──
8 print ( "Loading model..." )
9 processor = WhisperProcessor . from_pretrained (
10 "harphool17/whisper-large-v3-children-asr"
11 )
12 model = WhisperForConditionalGeneration . from_pretrained (
13 "harphool17/whisper-large-v3-children-asr"
14 )
15
16 # Use GPU if available (much faster!)
17 device = "cuda" if torch . cuda . is_available ( ) else "cpu"
18 model = model . to ( device )
19 model . eval ( )
20 print ( f"Model loaded on: { device } " )
21
22 # ── Step 2: Load Audio ──
23 def load_audio ( audio_path ) :
24 """Load audio file and convert to 16kHz mono"""
25 audio , sr = sf . read ( audio_path , dtype = "float32" )
26
27 # Convert stereo to mono if needed
28 if audio . ndim > 1 :
29 audio = audio . mean ( axis = 1 )
30
31 # Resample to 16kHz if needed
32 if sr != 16000 :
33 audio = librosa . resample ( audio , orig_sr = sr , target_sr = 16000 )
34
35 return audio
36
37 # ── Step 3: Transcribe ──
38 def transcribe ( audio_path ) :
39 """Transcribe children's speech from audio file"""
40
41 # Load audio
42 audio = load_audio ( audio_path )
43
44 # Process audio into model input
45 inputs = processor (
46 audio ,
47 sampling_rate = 16000 ,
48 return_tensors = "pt"
49 )
50
51 # Move to GPU if available
52 input_features = inputs . input_features . to ( device )
53
54 # Generate transcription
55 with torch . no_grad ( ) :
56 predicted_ids = model . generate (
57 input_features ,
58 language = "en" ,
59 task = "transcribe" ,
60 max_new_tokens = 225 ,
61 )
62
63 # Convert numbers back to text
64 transcription = processor . batch_decode (
65 predicted_ids ,
66 skip_special_tokens = True
67 ) [ 0 ]
68
69 return transcription . lower ( ) . strip ( )
70
71 # ── Step 4: Use It! ──
72 audio_file = "child_speech.wav" # Replace with your audio file path
73 result = transcribe ( audio_file )
74 print ( f"Child said: { result } " )
Supported Audio Formats
Format Extension Supported WAV .wav✅ Yes FLAC .flac✅ Yes MP3 .mp3✅ Yes OGG .ogg✅ Yes
Best format: WAV or FLAC (lossless quality)
Required: 16kHz sample rate, mono channel (code above handles this automatically!)
Common Errors and Fixes
Error Cause Fix ModuleNotFoundError: transformersLibrary not installed Run pip install transformers CUDA out of memoryGPU memory full Add model = model.to("cpu") RuntimeError: Input size mismatchWrong audio format Use the load_audio() function above OSError: model not foundWrong model name Check spelling: harphool17/whisper-large-v3-children-asr
📊 Training Details
Parameter Value Base Model OpenAI Whisper Large v3 Training Data 82,490 children's speech samples Training Hours ~185 hours of audio Age Groups 3-4, 5-7, 8-11 years Training Steps 2,576 steps Batch Size 4 (effective: 64 with gradient accumulation) Learning Rate 1e-5 Optimizer AdamW with cosine schedule Precision bfloat16 GPU 2x NVIDIA RTX 4500 Ada (24GB each)
📈 Performance
Split WER Description Validation 0.1030 10.3% word error rate Public Test 0.4432 Competition test set
Note: The gap between validation and public test WER is due to distribution shift — the test set contains more challenging audio conditions than the training data.
🔗 Related Resources
📝 Citation
If you use this model in your work, please cite:
1 @misc{whisper-children-asr-2026,
2 author = {Harphool Singh},
3 title = {Whisper Large v3 Fine-tuned for Children's Speech Recognition},
4 year = {2026},
5 publisher = {HuggingFace},
6 url = {https://huggingface.co/harphool17/whisper-large-v3-children-asr}
7 }
👤 Author
Harphool Singh
Built with ❤️ for improving children's education technology
