You can deploy this model and start earning money today!

soda-research/
p2-flat-d896-93cdf5a9

Name: p2-flat-d896-93cdf5a9
Availability: InStock

next-token-prediction

Views

No views yet

p2-flat-d896-93cdf5a9

One of 13 runs from the SODA-extension flat-vs-hier campaign: a compute-matched comparison of the flattened SODA architecture (paper) against a CSM/Moshi-style hierarchical factorization (backbone over steps + small depth transformer over the 8 Mimi RVQ codebook slots), trained on the same interleaved text+audio corpus with a shared id-space so held-out NLL is directly comparable across arms.

🌐 Project page: https://soda-audio.github.io
💻 Code + full study: https://github.com/potsawee/marin/tree/soda-extension/experiments/audio — results and analysis in report/FINDINGS.md
🗂️ All models: soda-research

Model details


Architecture	flattened (single decoder over the flat token stream)
Loss recipe	uniform cross-entropy over all tokens
Compute budget (3× forward FLOPs)	3e18
Model	d=896, L=9
Window	4096 tokens
Total parameters (incl. embeddings)	375M
Training step	48960
Audio	Mimi RVQ, 1 semantic + 7 acoustic codebooks, 12.5 Hz

Trained on the 42k-hour campaign corpus (YODAS + Emilia EN, interleaved text+audio, single pass).

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("soda-research/p2-flat-d896-93cdf5a9")
model = AutoModelForCausalLM.from_pretrained("soda-research/p2-flat-d896-93cdf5a9")