Zero-shot text-to-speech fine-tuned to produce natural Tanglish — casual code-mixed Tamil transliterated into the Latin alphabet — in the voice of a Chennai/Coimbatore speaker.

Fine-tune of fishaudio/fish-speech-1.5 using LoRA r=8, alpha=16 (0.96 % trainable params) on 24 hours of Tanglish audio from the companion dataset sugiv/tanglish-audio-v1.

Training metrics (verified)

Metric	Value
Base model	`fishaudio/fish-speech-1.5`
LoRA config	r=8, alpha=16, target=all-linear
Effective batch size	24 (12 x 2 grad accum)
Learning rate	cosine, warmup=200, peak=1e-4
Precision	bfloat16
Best val/loss	7.9294 (step 1,300)
Val/loss at start (step 100)	9.98
Relative improvement	-20 %
Early stop fired at	step ~1,800 (patience 8)
Training wall	9 h 5 min on 1x L40 48 GB
WandB run	sugi205/fish-speech-tanglish/9jcwux9l

Eval (Phase 2, 36 WAVs on L4 24 GB SECURE)

15 Tanglish sentences on default speaker (karthik_kumar_solo)
6 Tanglish sentences across the other 5 speakers (cross-speaker A/B)
15 identical sentences on stock fish-speech-1.5 (A/B baseline)
Reference clip: karthik_kumar_solo/Ow1q3_xSJWw_000000.wav (2.52 s)
Reference text: "innaiku vandhadhukku mannichitten. problem!"
Inference: temperature=0.7, top_p=0.7, repetition_penalty=1.5, seed=42, bf16
Throughput: ~20 tokens/s on L4, 163 s wall for 36 samples

Full eval writeup with human-listener protocol in eval/PHASE_2_EVAL.md. Raw WAVs under eval/{in_distribution,novel,hard,cross_speaker,ab_base}/.

Native-speaker verdict (2026-07-14): intelligible + usable Tanglish; prosody is somewhat robotic and there are minor audio-quality artifacts — expected for 0.96 % trainable LoRA on 24 h of data with only -20 % val_loss over the base model. See PHASE_2_EVAL.md for the detailed rubric.

Path	What
`model.pth`	Merged 1.28 GB LLM half of fish-speech (the text-to-semantic backbone). NOT the VQGAN vocoder — see below.
`config.json`	Model architecture config (Fish-Speech 1.5 dual-AR llama)
`tokenizer.tiktoken`	Tiktoken tokenizer (same as base)
`special_tokens.json`	Special-token map
`lora-ckpts/`	Raw Lightning checkpoints: `last.ckpt`, `step_1200/1300/1800.ckpt`. `step_1300` is the best-val checkpoint.
`eval/`	Phase 2 evaluation bundle: MANIFEST + 36 WAVs + write-up

What is missing (and why)

The VQGAN vocoder (firefly-gan-vq-fsq-8x1024-21hz-generator.pth) is not bundled here — it is unchanged from the base model. Download it once from fishaudio/fish-speech-1.5 and cache it locally.

Usage

python

import torch
from huggingface_hub import snapshot_download, hf_hub_download
from fish_speech.inference_engine import TTSInferenceEngine
from fish_speech.models.text2semantic.inference import launch_thread_safe_queue
from fish_speech.models.vqgan.inference import load_model as load_decoder_model
from fish_speech.utils.schema import ServeReferenceAudio, ServeTTSRequest

# 1. Tanglish LLM half (this repo)
llm_dir = snapshot_download("sugiv/fish-speech-1.5-tanglish", token="hf_...")

# 2. VQGAN vocoder (upstream, unchanged)
vq_ckpt = hf_hub_download(
    "fishaudio/fish-speech-1.5",
    "firefly-gan-vq-fsq-8x1024-21hz-generator.pth",
)

llama_queue = launch_thread_safe_queue(
    checkpoint_path=llm_dir, device="cuda", precision=torch.bfloat16, compile=False,
)
decoder = load_decoder_model(
    config_name="firefly_gan_vq", checkpoint_path=vq_ckpt, device="cuda",
)
engine = TTSInferenceEngine(
    llama_queue=llama_queue, decoder_model=decoder,
    precision=torch.bfloat16, compile=False,
)

ref_audio_bytes = open("ref.wav", "rb").read()
for r in engine.inference(ServeTTSRequest(
    text="machi, nalaikku Chennai la enna weather?",
    references=[ServeReferenceAudio(audio=ref_audio_bytes, text="innaiku vandhadhukku mannichitten. problem!")],
)):
    if r.code == "final":
        sr, audio_np = r.audio
        break

Known limitations

Prosody is somewhat wooden — LoRA r=8 alpha=16 has too few trainable params to fully overcome the base model's English + Mandarin prosody prior. Backlog item: retrain at r=32 alpha=64 with same corpus.
Requires an in-domain reference clip (2-4 s) — the model is a zero-shot TTS conditioned on a speaker prompt. Use one of the 6 speakers from sugiv/tanglish-audio-v1 for best results.
Not compatible with transformers.from_pretrained — fish-speech uses a bespoke Lightning-checkpoint format. Load via fish_speech.inference_engine.

License

CC-BY-NC-SA-4.0 (inherited from fishaudio/fish-speech-1.5). Non-commercial only.

Citation

bibtex

@misc{tanglish_fish_speech_2026,
  title={Fish-Speech 1.5 Tanglish LoRA},
  author={sugiv},
  year={2026},
  url={https://huggingface.co/sugiv/fish-speech-1.5-tanglish}
}

Companion audio dataset: sugiv/tanglish-audio-v1
Companion Qwen text agent: sugiv/qwen3-8b-tanglish
Base model: fishaudio/fish-speech-1.5

sugiv/
fish-speech-1.5-tanglish

Fish-Speech 1.5 · Tanglish LoRA (full_v1)