You can deploy this model and start earning money today!

soda-research/
p2-hier-d512-09a1dd5a

Name: p2-hier-d512-09a1dd5a
Availability: InStock

next-token-prediction

Views

No views yet

p2-hier-d512-09a1dd5a

One of 13 runs from the SODA-extension flat-vs-hier campaign: a compute-matched comparison of the flattened SODA architecture (paper) against a CSM/Moshi-style hierarchical factorization (backbone over steps + small depth transformer over the 8 Mimi RVQ codebook slots), trained on the same interleaved text+audio corpus with a shared id-space so held-out NLL is directly comparable across arms.

🌐 Project page: https://soda-audio.github.io
💻 Code + full study: https://github.com/potsawee/marin/tree/soda-extension/experiments/audio — results and analysis in report/FINDINGS.md
🗂️ All models: soda-research

Model details


Architecture	hierarchical (backbone over steps + depth transformer over codebook slots)
Loss recipe	Moshi-style alpha = 100/100/1 (text/semantic/acoustic)
Compute budget (3× forward FLOPs)	3e18
Backbone	d=512, L=6
Depth transformer	d=256, L=4
Window	1024 steps
Total parameters (incl. embeddings)	178M
Training step	56222
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

Loading

The hierarchical model ships its own modeling code (modeling_soda_hier.py, configuration_soda_hier.py) and loads with trust_remote_code:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p2-hier-d512-09a1dd5a", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("soda-research/p2-hier-d512-09a1dd5a", trust_remote_code=True)