You can deploy this model and start earning money today!

k1ngtai/
shan-mms-homhom

Name: shan-mms-homhom
Availability: InStock

tensorboard

safetensors

vits

Views

No views yet

Fine-tuned VITS Model for Shan Language (shn)

This repository contains a VITS (Variational Inference for Text-to-Speech) model fine-tuned for the Shan language (shn).

Project Overview

This project focuses on adapting the powerful VITS architecture to generate natural-sounding speech for the Shan language. Starting from a pre-trained multilingual model, we fine-tuned it on a custom Shan dataset to achieve improved pronunciation and intonation specific to Shan.

Model Description

Base Model: The fine-tuning process began with the facebook/mms-tts-shn model from Hugging Face, which provides a strong multilingual foundation.
Architecture: VITS (Variational Inference for Text-to-Speech), a neural network model for high-quality speech synthesis.
Language: Shan (shn)
Purpose: Converts written Shan text into spoken audio.

Fine-tuning Details

Dataset: A proprietary Shan language dataset comprising approximately 327 audio-text pairs was used for fine-tuning. This dataset was carefully curated to ensure data quality and relevance to the Shan language.
Training Framework: Hugging Face Transformers library, integrated with Accelerate for efficient distributed training.
Customizations & Key Steps:
1. Data Preprocessing: Handled specific CSV parsing challenges for metadata_wav.csv to correctly extract audio file paths and transcriptions.
2. Custom Dataset Builder: Implemented a custom ShanTTSDataset for seamless integration with the Hugging Face datasets library.
3. Tokenizer Expansion: Added missing Shan characters (e.g., ၷ, ၹ, ၻ, ၾ, ႊ, ရ, ႂ, ႆ) to the base model's tokenizer vocabulary to ensure comprehensive character coverage.
4. Embedding Resizing: Dynamically resized the model's token embeddings to accommodate the expanded vocabulary, initializing new embeddings with the mean of existing ones.
5. Configuration Management: Ensured preprocessor_config.json was correctly placed and updated model configuration files (e.g., vocab_size) to reflect tokenizer changes.
6. Training: Executed training with a custom configuration, including specific learning rates, batch sizes, and loss weights, for 100 epochs.

Usage

To generate speech using this fine-tuned model, you can use the transformers library:

python

from transformers import VitsModel, VitsTokenizer
import torch
import scipy.io.wavfile as wavfile

# Specify the path to your fine-tuned model directory
# If you've uploaded it to Hugging Face, you can use "your-hf-username/your-model-name"
MODEL_PATH = "/content/shan-mms-vits-output" # Example: local path after fine-tuning

# Load the model and tokenizer
model = VitsModel.from_pretrained(MODEL_PATH)
tokenizer = VitsTokenizer.from_pretrained(MODEL_PATH)

# Define the Shan text you want to synthesize
text = "မိူင်းတႆးၵေႃႈပႆႇပႃႈလႆႈၶိုၼ်း ပူၼ်ႉမႃးသိပ်းပီယဝ်ႉလူး…" # Example Shan text

# Tokenize the input text
inputs = tokenizer(text, return_tensors="pt")

# Generate the speech waveform
with torch.no_grad():
    # Note: ensure 'noise_scale', 'noise_scale_duration', 'length_scale' are not passed
    # if the model's forward method does not explicitly accept them.
    output = model(**inputs).waveform

# Save the generated audio to a WAV file
wavfile.write(
    "generated_shan_audio.wav",
    rate=model.config.sampling_rate,
    data=output[0].cpu().numpy() # Extract the first waveform and convert to numpy
)

print("Audio generated and saved to generated_shan_audio.wav")

# In a Colab environment, you can play the audio directly:
# from IPython.display import Audio, display
# display(Audio("generated_shan_audio.wav", autoplay=True))

Author

k1ngtai
- [Your GitHub Profile (k1ngtai)]
- [Your Hugging Face Profile (k1ngtai)]

License

This project is licensed under the [MIT License].

k1ngtai/shan-mms-homhom