You can deploy this model and start earning money today!

Kaushalb11/
seamless-s2t-finetune

Name: seamless-s2t-finetune
Availability: InStock

Views

No views yet

Direct Speech-to-Text Translation Fine-tuning

This repository contains scripts and resources for fine-tuning direct speech-to-text (S2T) translation models for English to Indian languages (Hindi, Gujarati, and Telugu) without intermediate text generation.

📁 Repository Structure

.
├── finetune_scripts/
│   ├── data_collect_clean.py      # Data collection and cleaning
│   ├── dataset_making.py           # Dataset preparation for fine-tuning
│   ├── finetune.py                 # Model fine-tuning script
│   ├── mt_eval.py                  # Evaluation and comparison script
│   ├── eval_finetuned/             # Evaluation results and predictions
│   ├── final-data/                 # Processed datasets (language-wise)
├──finetuned_models/           # Fine-tuned model checkpoints (.pt files)

🎯 Translation Tasks

Hindi → English (hin-eng)
Gujarati → English (guj-eng)
Telugu → English (tel-eng)

Note: All translations are performed using direct speech-to-text without intermediate text generation.

🚀 Pipeline Overview

1. Data Collection and Cleaning (`data_collect_clean.py`)

Downloads and cleans audio data from five AI4Bharat datasets:

ai4bharat/IndicVoices-ST
ai4bharat/NPTEL
ai4bharat/WordProject
ai4bharat/Spoken-Tutorial
ai4bharat/Mann-ki-Baat

Output: Cleaned dataset saved to Kaushalb11/multi-lang-s2t-dataset

2. Dataset Preparation (`dataset_making.py`)

Processes and structures the cleaned data for fine-tuning purposes.

Output: Language-wise datasets saved in final-data/ directory

3. Model Fine-tuning (`finetune.py`)

Fine-tunes the speech-to-text translation models for each target language.

Output: Model checkpoints saved as .pt files in finetuned_models/ directory

4. Evaluation (`mt_eval.py`)

Evaluates model performance by:

Generating baseline metrics (BLEU and chrF scores)
Loading fine-tuned model weights from .pt files
Comparing baseline vs fine-tuned model performance
Generating predictions

Output: Evaluation metrics and predictions saved in eval_finetuned/ directory

📊 Evaluation Metrics

BLEU Score: Measures translation quality based on n-gram precision
chrF Score: Character-level F-score for translation evaluation

🤗 Dataset

The cleaned dataset is available on Hugging Face:
Kaushalb11/multi-lang-s2t-dataset

📄 Output Files

`final-data/`

Contains processed datasets organized by target language:

Hindi (hin)
Gujarati (guj)
Telugu (tel)

`finetuned_models/`

Contains trained model checkpoints in PyTorch format (.pt files)

`eval_finetuned/`

Contains:

Baseline evaluation metrics
Fine-tuned model evaluation metrics
Comparison results
Translation predictions

🔬 Model Architecture

Direct speech-to-text translation without intermediate text generation, enabling:

Faster inference
Reduced error propagation
End-to-end optimization

🙏 Acknowledgments

LTRC, IIIT Hyderabad for providing GPU facilities and mentorship
AI4Bharat for providing the Indic language datasets

Kaushalb11/seamless-s2t-finetune