You can deploy this model and start earning money today!

Messam174/
speecht5_finetuned_essam2_ar

Name: speecht5_finetuned_essam2_ar
Availability: InStock

generated_from_trainer

MBZUAI/speecht5_tts_clartts_ar

finetune

mit

endpoints_compatible

Views

No views yet

speecht5_finetuned_essam2_ar

This model is a fine-tuned version of MBZUAI/speecht5_tts_clartts_ar on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.3333

Uses

🤗 Transformers Usage

You can run ArTST TTS locally with the 🤗 Transformers library.

First install the 🤗 Transformers library, sentencepiece, soundfile and datasets(optional):

pip install --upgrade pip
pip install --upgrade transformers sentencepiece datasets[audio]

Run inference via the Text-to-Speech (TTS) pipeline. You can access the Arabic SPeechT5 model via the TTS pipeline in just a few lines of code!

python

from transformers import pipeline
from datasets import load_dataset
import soundfile as sf

synthesiser = pipeline("text-to-speech", "("Messam174/speecht5_finetuned_essam2_ar")

embeddings_dataset = load_dataset("herwoww/arabic_xvector_embeddings", split="validation")
speaker_embedding = torch.tensor(embeddings_dataset[105]["speaker_embeddings"]).unsqueeze(0)
# You can replace this embedding with your own as well.

speech = synthesiser("السلام عليكم ورحمة الله وبركاته حياكم الله جميعا", forward_params={"speaker_embeddings": speaker_embedding})
# ArTST is trained without diacritics.

sf.write("speech.wav", speech["audio"], samplerate=speech["sampling_rate"])

Run inference via the Transformers modelling code - You can use the processor + generate code to convert text into a mono 16 kHz speech waveform for more fine-grained control.

python

from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan
from datasets import load_dataset
import torch
import soundfile as sf
from pydub import AudioSegment

# Check if GPU is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

# Load processor, model, and vocoder
processor = SpeechT5Processor.from_pretrained("Messam174/speecht5_finetuned_essam2_ar")
model = SpeechT5ForTextToSpeech.from_pretrained("Messam174/speecht5_finetuned_essam2_ar").to(device)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").to(device)

# Prepare inputs
inputs = processor(
    text="السلام عليكم ورحمة الله وبركاته حياكم الله جميعا", return_tensors="pt"
).to(device)

# Load xvector containing speaker's voice characteristics from a dataset
embeddings_dataset = load_dataset("herwoww/arabic_xvector_embeddings", split="validation")
speaker_embedding = torch.tensor(embeddings_dataset[105]["speaker_embeddings"]).unsqueeze(0).to(device)

# Generate speech
with torch.no_grad():  # Disable gradient computation for inference
    speech = model.generate_speech(inputs["input_ids"], speaker_embedding, vocoder=vocoder)

# Save the output as WAV
wav_file = "speech.wav"
sf.write(wav_file, speech.cpu().numpy(), samplerate=16000)
print(f"Speech saved to '{wav_file}'")

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 4
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 32
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 100
training_steps: 500
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.3806	0.3742	100	0.3452
0.3873	0.7484	200	0.3487
0.3788	1.1225	300	0.3441
0.3676	1.4967	400	0.3380
0.3668	1.8709	500	0.3333

Framework versions

Transformers 4.46.3
Pytorch 2.5.1+cu121
Datasets 3.2.0
Tokenizers 0.20.3