Wav2Vec2-XLS-R-300m finetuned on the data of Korean pronunciations of English speakers.

Data Information

• Dataset Name: the spoken Korean voice of native English speakers.
• Data Type: Speech recordings of English speakers speaking Korean.
• Annotation: Each utterance is annotated with korean words and phoneme sequences.
• Train Set: 124,626 samples, 121.75 hours
• Valid Set: 15,066 samples, 14.94 hours
• Test Set: 15,091 samples, 14.78 hours

Training Procedure

1. Data preprocessing to align audio with phoneme labels.
2. Wav2Vec2-XLS-R-300M model fine-tuning with CTC loss.
3. Evaluation on validation and test sets.

Training Hyperparameters

• Epochs: 50
• Learning Rate: 0.0001
• Warmup Ratio: 0.1
• Scheduler: Linear
• Batch Size: 8
• Loss Reduction: Mean
• Feature Extractor Freeze: Enabled

Training Results

• Final Training Loss: 0.0473
• Validation Loss: 0.1541
• Word Error Rate (WER) on Validation Set: 0.1156

Test Results

• Word Error Rate (WER): 0.0315
• Character Error Rate (CER): 0.0230
• Phoneme Error Rate (PER): 0.0315

Phoneme Data Example

• Prompt means the specific sentence or word that the pariticipant has to read.
• True Phonemes indicate the phonemes transcribed based on how the participant's pronunciation sounds
• Predicted Phonemes represent the finetuned model's prediction of phonemes.

• Prompt: 여기 비빔밥 한 그릇 갖다 주세요.
• True Phonemes of Korean pronunciation: ㅕ ㄱ ㅣ ㅂ ㅣ ㅂ ㅣ ㅁ ㅂ ㅏ ㅂ ㅎ ㅏ ㄴ ㄱ ㅡ ㄹ ㅡ ㄷ ㄱ ㅏ ㄷ ㅏ ㄷ ㅗ ㅅ ㅔ ㅛ
• Predicted Phonemes: ㅕ ㄱ ㅣ ㅂ ㅣ ㅂ ㅣ ㅁ ㅃ ㅏ ㅂ ㅎ ㅏ ㄴ ㄱ ㅡ ㄹ ㅡ ㄷ ㄱ ㅏ ㄷ ㄸ ㅏ ㅈ ㅜ ㅅ ㅔ ㅛ

Training Logs

• events.out.tfevents.1741331703.oem-WS-C621E-SAGE-Series.3197499.0
• events.out.tfevents.1741696761.oem-WS-C621E-SAGE-Series.3197499.1