You can deploy this model and start earning money today!

multilingual-tts/
F5-TTS-OpenBible-Malayalam

Name: F5-TTS-OpenBible-Malayalam
Availability: InStock

davidguzmanr/open-bible-resources

SWivid/F5-TTS

finetune

cc-by-sa-4.0

Views

No views yet

F5-TTS Open Bible — Malayalam

A zero-shot text-to-speech model for Malayalam, trained from scratch on the Open Bible corpus using the F5-TTS architecture (diffusion transformer with vocos vocoder, 24 kHz output).

The model takes a short reference audio clip (5–10 seconds) and a target text, and synthesises the target text in the voice of the reference speaker. No fine-tuning per voice is required.

Files

File	Purpose
`model_last.pt`	Trained model weights.
`vocab.txt`	Character vocabulary built from the training transcripts.
`F5-TTS_OpenBible_Malayalam.yaml`	Hydra training/inference config (architecture, mel spec settings, tokenizer).

Intended use

Zero-shot TTS for Malayalam, controlled by a user-supplied reference clip.
Research on multilingual TTS, low-resource TTS evaluation, and listening studies on Open Bible–style read-speech.

How to use

Install F5-TTS:

pip install git+https://github.com/SWivid/F5-TTS.git

Download the checkpoint and run inference:

python

import torch
from huggingface_hub import hf_hub_download
from hydra.utils import get_class
from omegaconf import OmegaConf
from f5_tts.infer.utils_infer import infer_process, load_model, load_vocoder, preprocess_ref_audio_text

repo_id = "multilingual-tts/F5-TTS-OpenBible-Malayalam"
ckpt   = hf_hub_download(repo_id, "model_last.pt")
vocab  = hf_hub_download(repo_id, "vocab.txt")
config = hf_hub_download(repo_id, "F5-TTS_OpenBible_Malayalam.yaml")

device = "cuda" if torch.cuda.is_available() else "cpu"

model_cfg = OmegaConf.load(config)
model_cls = get_class(f"f5_tts.model.{model_cfg.model.backbone}")

vocoder = load_vocoder(vocoder_name="vocos", is_local=False, device=device)
model   = load_model(
    model_cls, model_cfg.model.arch, ckpt,
    mel_spec_type="vocos", vocab_file=vocab, use_ema=True, device=device,
)

# Supply your own clean reference clip — 5–10 s, single speaker and its transcription.
ref_audio = "/path/to/your-malayalam-clip.wav"
ref_text  = "Exact transcription of the clip"
gen_text  = "..."   # text to synthesise in Malayalam

ref_audio_proc, ref_text_proc = preprocess_ref_audio_text(ref_audio, ref_text)
wav, sr, _ = infer_process(
    ref_audio_proc, ref_text_proc, gen_text, model, vocoder,
    mel_spec_type="vocos", device=device,
)

Training data

Source: davidguzmanr/open-bible-resources, config Malayalam
Size: approximately 24,581 utterances
Speakers: multispeaker; speaker identity is supplied at inference time via the reference clip, not by a fixed speaker id
Sample rate: 24 kHz
Maximum utterance duration during training: 15 s

Training procedure

Base architecture: F5-TTS v1 Base (DiT, 1024 dim, 22 layers, 16 heads, text dim 512, 4 convolutional layers).
Tokenizer: custom character-level, built from the training transcripts.
Vocoder: vocos.
Mel spectrogram: 100 channels, hop 256, win 1024, n_fft 1024.
Optimizer: AdamW, learning rate 7.5e-5, 20 000 warmup updates.
Training budget: 500,000 optimizer updates on 4 GPUs with mixed precision (bf16), global batch ≈ 112,000 frames.

Audio preprocessing, vocab generation, and config sizing are reproducible via the upstream open-bible-models repo.

Evaluation

Evaluated alongside other Open-Bible TTS systems on character/word error rate (via Meta's Omnilingual ASR) and UTMOSv2 naturalness scores. See the open-bible-models repository for the evaluation pipeline and the open-bible-surveys repository for the human-listening survey methodology.