You can deploy this model and start earning money today!

soda-research/
p1-hier-moshi-765a7592

Name: p1-hier-moshi-765a7592
Availability: InStock

next-token-prediction

Views

No views yet

p1-hier-moshi-765a7592

One of 13 runs from the SODA-extension flat-vs-hier campaign: a compute-matched comparison of the flattened SODA architecture (paper) against a CSM/Moshi-style hierarchical factorization (backbone over steps + small depth transformer over the 8 Mimi RVQ codebook slots), trained on the same interleaved text+audio corpus with a shared id-space so held-out NLL is directly comparable across arms.

🌐 Project page: https://soda-audio.github.io
💻 Code + full study: https://github.com/potsawee/marin/tree/soda-extension/experiments/audio — results and analysis in report/FINDINGS.md
🗂️ All models: soda-research

Model details


Architecture	hierarchical (backbone over steps + depth transformer over codebook slots)
Loss recipe	Moshi-style alpha = 100/100/1 (text/semantic/acoustic)
Compute budget (3× forward FLOPs)	3e18
Backbone	d=768, L=8
Depth transformer	d=384, L=4
Window	1024 steps
Total parameters (incl. embeddings)	308M
Training step	56570
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

Loading

The hierarchical model ships its own modeling code (modeling_soda_hier.py, configuration_soda_hier.py) and loads with trust_remote_code:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p1-hier-moshi-765a7592", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/p1-hier-moshi-765a7592", trust_remote_code=True)