Nanospeech is a research-oriented project to build a minimal, easy to understand text-to-speech system that scales to any level of compute. It supports voice matching from a reference speech sample, and comes with a variety of different voices built in.

An 82M parameter pretrained model (English-only) is available, which was trained on a single H100 GPU in a few days using only public domain data. The model is intentionally small to be a reproducible baseline and allow for fast inference. On recent M-series Apple Silicon or Nvidia GPUs, speech can be generated around ~3-5x faster than realtime.

All code and pretrained models are available under the MIT license, so you can modify and/or distribute them as you'd like.

Nanospeech is based on a current line of research in text-to-speech systems which jointly learn text alignment and waveform generation. It's designed to use minimal input data — just audio and text — and avoid any auxiliary models, such as forced aligners or phonemizers.

There are two single-file implementations, one in PyTorch and one in MLX, which are near line-for-line equivalence where possible to make it easy to experiment with and modify. Each implementation is around 1,500 lines of code.

Use the --voice parameter to select the voice used for speech:

celeste — Sample

luna — Sample

nash — Sample

orion — Sample

rhea — Sample

Note these voices are all based on samples from the LibriTTS-R dataset.

You can also provide a speech sample and a transcript to match to a specific voice, although the pretrained model has limited voice matching capabilities. See python -m nanospeech.generate --help for a full list of options to customize the voice.

Nanospeech includes a PyTorch-based trainer using Accelerate, and is compatible with DistributedDataParallel for multi-GPU training.

It supports streaming from any WebDataset, but it should be straightforward to swap in your own dataloader as well. An ideal dataset consists of high-quality speech paired with clean transcriptions.

See the examples for an example of training both the base model and the duration predictor on the large-scale Emilia dataset (note: Emilia is CC BY-NC-4.0 licensed).

As a research project, the pretrained model that comes with Nanospeech isn't designed for production usage. It may mispronounce words, has limited capability to match out-of-distribution voices, and can't generate very long speech samples.

However, the underlying architecture should scale up well to significantly more compute and larger datasets, so if training your own model is attractive, you can extend it to perform high-quality voice matching, multilingual speech generation, emotional expression, etc.