- The glottal source is the vibration of the vocal folds themselves — the raw periodic pulse train that becomes perceived pitch.
- The vocal-tract filter is the resonant shaping applied by the throat, mouth cavity, and tongue position. This is the component that makes a voice sound like a specific individual, and that makes a given vowel sound like that vowel regardless of who's producing it.
- The turbulent noise component captures breath, fricative consonants, and the non-tonal texture present in both speech and singing — the airy quality of a breathy delivery, the hiss of an "s" sound.
Rather than hoping a neural network discovers this structure implicitly through training, TriStream builds it directly into the architecture as a hard constraint.
Full Architecture Breakdown
The Three Parallel Encoder Streams
The heart of TriStream is a set of three encoders that process the same underlying audio event from entirely separate vantage points, with no ability to exchange information until much later in the network.
Stream 1 — Source (3 transformer layers). This stream receives only two numeric inputs: a log-scaled F0 pitch contour and a binary voiced/unvoiced flag. Critically, it never receives the mel spectrogram, the singer embedding, or any timbre-carrying signal whatsoever. It is architecturally incapable of representing anything about who is singing — it can only ever describe what pitch is being produced. This is what makes the disentanglement structural rather than merely encouraged: there is no pathway through which timbre information could reach this stream, even if training somehow pushed the model in that direction.
Stream 2 — Filter (5 transformer layers, cross-attending to the lyric encoder). This is the deepest of the three streams, reflecting that formant structure and timbre are the hardest components of a voice to model well. It receives the mel spectrogram along with a singer identity embedding and an emotion/technique embedding, and cross-attends against the lyric representation so it always knows what phoneme is currently being produced. This is the only stream that carries identity information.
Stream 3 — Residual (2 transformer layers). Handles breath noise, fricative consonants, and general articulation texture — the parts of a vocal performance that aren't well described by either clean pitch or clean timbre.
The Lyric / Phoneme Encoder
A 4-layer transformer that processes input lyrics independently, converting them into a representation that conditions the Filter stream and, later, the Fusion Trunk, via cross-attention. This keeps "what should be sung" cleanly separated from "how it should sound" throughout most of the network.
The Late Fusion Trunk
An 8-layer transformer — again using cross-attention against the lyric encoder — where the three separately-processed streams are finally allowed to combine into one unified representation. This fusion point is deliberately placed as late as possible in the network. Everything before it has been kept artificially separated; this is the single location where pitch, timbre, and texture information are permitted to interact.
The Adversarial Singer Head
Attached to the output of the Fusion Trunk is a small classifier network trained to guess which singer produced a given piece of audio, using only the shared (fused) representation as input. Its gradient passes through a gradient reversal layer before propagating back into the trunk — meaning that as this classifier becomes better at correctly identifying singers, the trunk is pushed in the opposite direction, actively degrading its own ability to be identified by that classifier.
The practical effect is a shared representation that is explicitly trained to become as uninformative about singer identity as possible. It is pushed toward encoding general vocal mechanics — how a given formant transition behaves, how a phoneme boundary sounds — rather than person-specific fingerprints that could be exploited to memorize particular singers. This is a form of domain-adversarial training, applied here specifically as an anti-memorization mechanism rather than for its more typical use in domain adaptation.
The Speaker Encoder (Zero-Shot Voice Cloning)
A separate convolutional network handles the task of extracting a portable "voice fingerprint" from an arbitrary reference clip — including from voices the model has never encountered during training. It uses dilated convolutions to build a wide receptive field cheaply, squeeze-excitation blocks to let the network weight which channels matter most, and attentive statistics pooling (weighted mean and standard deviation, rather than a simple average) to collapse a variable-length audio clip into a single fixed-size embedding.
This encoder is trained with a contrastive objective: two different clips from the same speaker are pulled together in embedding space, while clips from different speakers are pushed apart. This is what gives the resulting embedding space genuine, learned structure — rather than an arbitrary set of numbers, it becomes a space where "closeness" reliably corresponds to "sounds like the same person," which is the property that makes zero-shot cloning of unseen voices possible at all.
Duration and Pitch Predictors
When generating an entirely new performance (as opposed to editing existing audio), the model has no ground-truth timing or melody to read from — it has to invent both.
The Duration Predictor estimates how many audio frames each phoneme should occupy, predicted in logarithmic space specifically because real note durations are heavily skewed: a sustained vowel might last dozens of times longer than a short consonant, and predicting in log-space prevents the long notes from dominating the training signal.
The Pitch Predictor deliberately avoids outputting one blended pitch curve. Instead, it separately predicts a base pitch (the target note itself), a "scoop" component (the natural human tendency to slide into a pitch rather than hit it instantaneously, which is one of the clearest tells separating human singing from robotic-sounding synthesis), and vibrato depth and rate. Vibrato is deliberately ramped in gradually over roughly a quarter-second, rather than snapping on instantly, mirroring how real vibrato actually builds over the course of a sustained note. Because these components are generated separately, they remain independently adjustable after the fact — vibrato intensity can be dialed up or down without disturbing underlying pitch accuracy, for instance.
Beat Alignment
When a backing track accompanies a generation request, a beat-tracking step extracts tempo and downbeat positions from that track, and the model's predicted note durations are pulled toward that rhythmic grid using soft quantization rather than a hard mechanical snap. Perfectly quantized timing is one of the most reliable tells of synthetic vocals; TriStream's alignment allows a controlled degree of natural deviation from the grid, including an adjustable "swing" parameter, specifically to avoid that mechanical quality.
The Flow-Matching Decoder
The final generative stage is a 5-layer decoder built around rectified flow matching, a more modern and computationally efficient alternative to the diffusion-based approaches used by earlier singing synthesis systems. Rather than learning to slowly reverse a noise-adding process across 100 or more denoising steps, the model learns to predict a direct velocity field pointing from random noise toward the target mel spectrogram. This allows full-quality audio generation in roughly 32 sampling steps — a substantial efficiency improvement over the diffusion approach used in systems like the original DiffSinger, while maintaining comparable output fidelity.
Two Modes, One Underlying Model
Editing is the model's native training objective — not a feature added on top of a generation-only system. During training, random contiguous spans of real audio are masked out, and the model is trained to reconstruct exactly that masked span using the surrounding, unmasked audio as context, along with the lyrics for that section. This means that "regenerate this one phrase with different lyrics" or "make this section sound more breathy" is not a special-cased inference trick bolted onto the architecture afterward — it is the literal task the network was designed and trained around from its very first training step.
Generation mode reuses every one of the same trained components, but begins from a fully masked target rather than a partially masked one: the Speaker Encoder clones the requested voice from reference audio, the Duration and Pitch Predictors invent complete timing and melody from the lyrics (and any supplied melody guide), and the Decoder generates the entire performance from scratch rather than filling a gap in existing material.
Training Methodology
Singer-disjoint validation. A held-out set of singers is determined by a stable hash function applied to singer identity, and that set is completely excluded from training data at the point of data loading — not merely reserved after the fact. Every reported validation score throughout training is measured exclusively against voices the model has never once encountered during training. This is meaningfully stricter than the more common approach of a random validation split, which can silently allow the same singer's recordings to appear in both the training set and the validation set, producing validation scores that look strong but don't actually measure generalization. The singer-disjoint approach is the only way to be confident the model is learning transferable vocal mechanics rather than quietly memorizing the specific individuals present in its training data.
Crash-resilient, autonomous checkpointing. The training pipeline automatically pushes model checkpoints to a remote Hugging Face repository whenever the singer-disjoint validation score improves on its previous best, and additionally on a rolling time-based interval regardless of whether an improvement occurred. If a training session is interrupted for any reason — a platform session limit, an infrastructure failure, a manual restart — the next invocation of the training script automatically detects the most recent available checkpoint, including full optimizer state, and resumes training from precisely that point rather than restarting from initialization. This infrastructure was built and hardened specifically in response to real interruptions encountered during development, including data-loader failures under multiprocessing and transient Hub connectivity issues, both of which are now handled automatically with retry logic and graceful fallback.
Careful data curation. Training data is deliberately restricted to dry, isolated vocal recordings, with instrumental bleed, heavy reverb, and mixed-track audio explicitly excluded — because mixed audio actively corrupts the mel-spectrogram reconstruction target the model trains against, regardless of how much of it is fed in. The training corpus combines large-scale speech data, used to teach general vocal-tract mechanics and broad phonetic coverage, with smaller, carefully verified singing corpora used specifically for pitch behavior, vocal technique, and musical phrasing. Datasets are individually probed and verified before inclusion — the pipeline tests that each configured data source actually loads and decodes correctly before committing to using it in a training run, rather than assuming configuration correctness.
Deliberate anti-overfitting measures, beyond the adversarial disentanglement described above:
- Waveform-level augmentation, including pitch shifting, time stretching, and gain variation, applied during training so the model never sees the exact same acoustic realization of a given clip twice
- SpecAugment-style frequency and time masking applied directly to the mel spectrogram during training
- Mixup applied between training samples within a batch, discouraging the model from latching onto exact spectral fingerprints of individual recordings
- Early stopping keyed directly to the singer-disjoint validation score, with patience-based tolerance for normal step-to-step noise — training halts automatically if that score fails to improve across a sustained run of evaluations
- The best-performing checkpoint by validation score, not simply the most recently trained one, is what gets preserved and deployed — these are tracked and saved as entirely separate files throughout training
Technical Specifications
| Property | Value |
|---|
| Total parameters | 321.8M |
| Architecture family | Custom transformer, source-filter factored |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm |
| Feed-forward network | SwiGLU |
| Attention mechanism | Scaled dot-product attention with cross-attention conditioning |
| Generative paradigm | Rectified flow matching |
| Typical inference sampling steps | ~32 |
| Sample rate | 24kHz (architecture supports scaling to 44.1kHz for full-band fidelity) |
| Mel spectrogram resolution | 100 mel bands (scales to 128 at higher sample rates) |
| Speaker conditioning | Zero-shot, via dedicated contrastive speaker encoder |
| Validation methodology | Singer-disjoint held-out evaluation set |
| Checkpointing | Automatic, crash-resilient, Hub-synchronized |
| Training precision | bf16 mixed precision on supported hardware |
Development Status
TriStream-SVS is actively in training. The complete architecture and pipeline are functioning end-to-end: data ingestion and verification, the three-stream encoder system, adversarial disentanglement, the speaker encoder, duration and pitch prediction, flow-matching generation, and fully automated crash-safe checkpointing are all live and operating together. Validation loss measured against held-out, previously unseen singers has continued to improve across the course of training, which is the primary signal the development process is tracking to confirm the model is learning genuinely transferable singing ability rather than memorizing its training data.
Architecture, training infrastructure, data pipeline, and all supporting tooling designed and built entirely from scratch by a solo developer at CorX Labs, Jamaica.
made in jamaica Corx Labs by Nathan
