Native training step-zero initialization for the 100M 10-prediction model. This exactly uses origin seed 1110 and historical source commit 36b3287e5d6a59c81d49eb92769e4a9fb571d8e8, matching the start of W&B run 6nf8goau. The weights are untrained FP32 values produced before XLA/FSDP wrapping.

Model Details

Parameter	Value
Parameters	~0.15B
Layers	12
Hidden Size	768
Attention Heads	12
Vocab Size	8192
Prediction Steps	10

Usage

python

from transformers import AutoModel, AutoConfig

# Load with trust_remote_code for custom model
model = AutoModel.from_pretrained(
    "TuKoResearch/AuriStream100M_10Pred_BigAudioDataset_500k-randinit",
    trust_remote_code=True,
)

# Or load config first
config = AutoConfig.from_pretrained("TuKoResearch/AuriStream100M_10Pred_BigAudioDataset_500k-randinit", trust_remote_code=True)

Base Model Code

This checkpoint uses shared model code from TuKoResearch/AuriStream-base.

Tokenizer

This model uses cochlear tokens from WavCochCausalV8192.