This repository hosts an updated, fine-tuned version of openai/whisper-small engineered to translate spoken Korean audio directly into English text transcripts.

Unlike V1 which only adapted self-attention mechanisms, this V2 model unfreezes encoder-decoder cross-attention mapping layers, resulting in a 3x translation baseline improvement and successful alignment of domain-specific sports terminology (e.g., baseball pitching terminology like slider and splitter).

Model Evaluation Performance

V2 Cross-Attention Fine-Tuned Model BLEU: 14.73
Original Baseline Whisper-Small BLEU: 4.64

Training Hyperparameters & Infrastructure

Hardware: NVIDIA A100 GPU
Quantization: 4-bit NormalFloat (nf4) with double quantization
LoRA Configurations: Rank ((r)) = 32, Alpha ((\alpha)) = 64
Target Modules: ["q_proj", "v_proj", "out_proj", "k_proj", "fc1", "fc2"] (Cross-Attention Enabled)
Learning Rate: 5e-5 (Stabilized)
Precision: Native Mixed Precision (bf16)
Optimization Batching: Effective Batch Size = 32 (Per Device Train Batch = 8, Gradient Accumulation Steps = 4)

didiudom94/whisper-small-ko-to-en-v2-cross-attention

Whisper Small - Korean to English Audio Translator (V2 Cross-Attention Optimized)

Model Evaluation Performance

Training Hyperparameters & Infrastructure

didiudom94/
whisper-small-ko-to-en-v2-cross-attention