Arabic Turn Detector Fine-Tuning - Data Collection, Preprocessing and Fine-Tuning

📋 Objective

📁 Project Structure

Hams's_Task/
├── notebook/
    ├── data_ingestion.ipynb          # Data collection and fucntions for labeling the data correctly
    ├── data_preprocessing.ipynb      # Data Preprocessing and EDA on the collected data
├── fine_tuning/
    ├── distilbert-base-multilingual-cased.py      # Fine-Tuned Distillbert model
    ├── fine_tuning_SmolLM2.py                     # Fine-Tuned SmolLM2 (Choosed as its performace were good)
    ├── SmolLM2_Artifacts                          # Contain the artifacts of SmolLM2 model
    ├── distillbert_artifacts                      # Contain the Confusion Metric image of Distillbert
├── README.md
├── raw_data/
    ├── dataset.csv                   # This is the dataset taken from Github
│   ├── labeled_data.csv              # Initial processing operation performed on the GitHub dataset like removing unnecessarily columns etc and the dataset by the name of "labeled_data.csv".
│   ├── yt_saudi_turns.csv           # Collected YouTube transcripts data (Saudi Conversational)
│   ├── final_raw_data.csv           # Combined initial datasets (labeled_data.csv + yt_saudi_turns.csv)
│   ├── correct_labeled_data.csv     # Fixed labeling errors on the "final_raw_data.csv"
│   ├── preprocessed_data.csv        # Take the "correct_labeled_data.csv" and perform basic preprocessing and EDA
│   ├── manual_data_collection/
│   │   └── manual_data.csv          # Manually collected data
│   └── clean_data/
│       └── processed_data.csv       # Take the "manual_data.csv" and "preprocessed_data.csv" and make the final cleaned dataset
└── model_data/                      # Take the "processed_data.csv" and split into trainig and validation
    ├── training_data.csv            # Training split (80%)
    └── validation_data.csv          # Validation split (20%)

🔄 Data Processing Pipeline

Step 1: Initial Data Collection

• ✅ Downloaded conversational Arabic dataset from GitHub
• ✅ Removed unnecessary columns
• ✅ Applied custom labeling function to:
  • Extract utterances from conversation columns
  • Create segments from each utterance
  • Label segments as "turn-end" or "not-turn-end"
  • Treat each utterance independently
• Output: raw_data/labeled_data.csv

Step 2: YouTube Data Collection

• ✅ Collected additional data from YouTube video transcripts
• ✅ Applied custom function with specific rules for turn-end detection
• ✅ Increased dataset size for better model performance
• Output: raw_data/yt_saudi_turns.csv

Step 3: Data Combination

• ✅ Combined YouTube data with initial GitHub data
• ✅ Created larger unified dataset
• Output: raw_data/final_raw_data.csv

Step 4: Label Correction

• ✅ Fixed incorrectly labeled text segments
• ✅ Applied custom function with specified correction rules
• ✅ Improved data quality and accuracy
• Output: raw_data/correct_labeled_data.csv

Step 5: Basic Preprocessing

• ✅ Applied standard text preprocessing techniques
• ✅ Cleaned and normalized text data
• Output: raw_data/preprocessed_data.csv

Step 6: Manual Data Integration

• ✅ Added manually collected data from raw_data/manual_data_collection/manual_data.csv
• ✅ Combined with preprocessed data for final dataset
• Output: raw_data/clean_data/processed_data.csv

Step 7: Train/Validation Split

• ✅ Split final dataset into training and validation sets
• ✅ Prepared data for model training
• Output:
  • model_data/training_data.csv
  • model_data/validation_data.csv

📊 Data Statistics

🛠️ Custom Functions Used

1. Initial Labeling Function

• Extracts utterances from conversation columns
• Creates text segments
• Labels each segment for turn-end detection

2. YouTube Processing Function

• Processes video transcript data
• Applies turn-end detection rules
• Formats data consistently

3. Label Correction Function

• Identifies and fixes mislabeled segments
• Applies correction rules
• Improves dataset quality

4. Preprocessing Function

• Cleans and normalizes text
• Prepares data for model training
• Handles Arabic text specifics ========================================================================

End-of-Utterance (EOU) Detection Model Comparison

📊 Model Overview

1. DistilBERT-Base-Multilingual-Cased (10 epochs)
2. SmolLM2-135M (5 epochs)

🎯 Task Description

• Class 0 (No EOU): The text does not represent the end of an utterance
• Class 1 (EOU): The text represents the end of an utterance

🤖 Model 1: DistilBERT-Base-Multilingual-Cased

Training Progress

Final Evaluation Results

Per-Class Performance

Confusion Matrix

Model Analysis

Model Analysis

⚠️ Issue Identified: The validation loss starts increasing after epoch 4 while training loss continues to decrease, indicating overfitting. Although accuracy and other metrics improve, the model is not generalizing well to unseen data.

🚀 Model 2: SmolLM2-135M (Recommended)

Training Progress

Final Evaluation Results

Per-Class Performance

Confusion Matrix

🧪 Test Results Comparison

Sample Predictions

📈 Performance Summary

🏆 Conclusion

1. Better Generalization: No signs of overfitting with consistently decreasing validation loss
2. Slightly Better Performance: Marginal improvement in all metrics
3. More Reliable: Stable training behavior and consistent predictions
4. Efficiency: Achieved comparable results with fewer epochs (5 vs 10)

🔧 Usage