You can deploy this model and start earning money today!

Views

No views yet

CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1

Name: CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1
Availability: InStock

CascaMini-350M-C24 is a 339,788,800-parameter base language model trained from scratch for 15,000,010,752 tokens.

This is the first production CascaMini-350M release. It is a pretrained base model, not an instruction-tuned or chat-aligned assistant.

Model summary

Property	Value
Parameters	339,788,800
Decoder layers	24
Hidden size	1,024
Query heads	16
KV heads	4
Head dimension	64
SwiGLU intermediate	3,072
Context length	2,048
Vocabulary	49,152
RoPE theta	10,000
Embedding / LM head	tied
Bias / dropout	no / 0
Architecture family	Llama-style decoder-only

The Hugging Face repository is a parity-checked conversion of the accepted custom CascaMini release to standard Transformers LlamaForCausalLM format.

Tokenizer

The model uses the SmolLM2-49K / cosmo2-tokenizer vocabulary.

The original tokenizer vocabulary and merges are retained. The Hugging Face model_max_length metadata is set to 2,048 to match the model's actual trained context length.

Training

Data mixture — CascaMini-CoreMix-v1 / M3

Source	Weight
FineWeb-Edu	55.00%
Wikimedia / Wikipedia	10.00%
peS2O	8.00%
arXiv	5.00%
Stack v2 code	12.00%
Math StackExchange	5.00%
LibreTexts	0.75%
Pressbooks	2.00%
OER Commons	0.75%
DOAB	0.50%
Structured Stack v2	1.00%

The CoreMix source documents were filtered and materialised before training. The final training sampler used the frozen M3 source weights above.

See TRAINING_DATA_AND_ATTRIBUTION.md for the retained source revisions, licence/provenance audit, and third-party data notes.

Optimizer

The production optimizer was VO Muon: Muon applied only to the attention value and output projection matrices, with auxiliary AdamW for the remaining parameters.

Muon:

peak LR: 0.0025
minimum LR: 0.00025
cosine schedule
warmup: 0
momentum: 0.95
Newton-Schulz iterations: 5
Muon weight decay: 0

Auxiliary AdamW:

peak LR: 2e-4
minimum LR: 2e-5
warmup: 100 steps
weight decay: 0.1
betas: (0.9, 0.95)
eps: 1e-8

Training used 8 GPUs, BF16, context length 2,048, and 65,536 tokens per global update.

Reported training throughput was approximately 512,637 tokens/s and the trainer-reported elapsed time was approximately 9.31 hours.

Final evaluation

The final checkpoint was also the best fixed-validation checkpoint.

Metric	Final value
Final validation loss	2.806850
FineWeb BPB	0.88373602
CoreMix raw-weighted BPB	0.79967575
All raw-weighted BPB	0.80351910
M3 source-weighted BPB	0.83542681

Per-source BPB

Source	BPB
FineWeb-Edu	0.883736
Wikimedia	0.906903
peS2O	0.802241
arXiv	0.809891
Stack v2 code	0.573321
Math StackExchange	0.879759
LibreTexts	0.651555
Pressbooks	0.856434
OER Commons	0.740656
DOAB	0.817797
Structured Stack v2	0.956214

3B -> 15B scaling

Using the same production-selected VO-Muon recipe:

Metric	~3B	15B	Change
M3 weighted BPB	0.894323	0.835427	-6.59%
FineWeb BPB	0.939407	0.883736	-5.93%
Validation loss	2.992783	2.806850	improved
L24 residual RMS	11.302	14.982	increased

Repetition stress tests

Long continuation repetition remains the clearest known weakness.

Generation mode	repeat2
Greedy	0.814915
Sampled (temperature 0.8, top-k 50, top-p 0.95, RP 1.05)	0.196018
Showcase (temperature 0.65, top-k 40, top-p 0.90, RP 1.08)	0.332300
Sampled RP 1.12	0.106421

These are deliberately long-generation stress diagnostics, not downstream task scores.

No standard downstream benchmark suite is claimed in this release yet.

Geometry diagnostics

Final diagnostic values:

L24 residual RMS: 14.9823
logit RMS: 6.4040
mean Q stable rank: 18.28
mean K stable rank: 13.75
mean V stable rank: 76.87
mean O stable rank: 123.80

The VO-Muon training line remained numerically stable through the full 15B schedule.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "beardymcgee/CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    device_map="auto",
)

prompt = "The most important property of a reproducible scientific experiment is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=96,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.05,
)

print(tokenizer.decode(output[0], skip_special_tokens=False))

For raw greedy decoding:

python

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)

Intended use

This release is intended for:

language-model research;
evaluation and reproducibility work;
fine-tuning / adaptation experiments;
studying small-model data, optimizer, and scaling behaviour.

It is not presented as a factual authority, safety-aligned assistant, or production decision system.

Limitations

This is a small base model. It can:

hallucinate or state false information confidently;
repeat phrases or enter repetitive continuation patterns;
generate biased, offensive, unsafe, or otherwise undesirable text;
reproduce fragments resembling training material;
perform poorly on tasks requiring knowledge or reasoning beyond its scale;
fail to follow instructions because it was not instruction-tuned.

The long-generation repetition measurements above are published explicitly because repetition remains a known weakness of this model family.

Training-data provenance and licensing

The canonical CoreMix JSONLs retained licence_evidence_mode and licence_fields for 100% of the 6,699,937 CoreMix training records audited for this release.

The corpus nevertheless contains multiple third-party licence classes, including CC BY, CC BY-SA, public-domain/CC0/GFDL material, and source code under many permissive SPDX licences. FineWeb-Edu is tracked separately through its upstream dataset terms.

For that reason this model card uses:

license: other

rather than implying that one simple permissive licence accurately describes all provenance and downstream rights questions associated with the release.

See MODEL_RELEASE_NOTICE.md and TRAINING_DATA_AND_ATTRIBUTION.md.

No training dataset is redistributed in this model repository.

Release integrity

Original accepted release identity:

CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1

Original accepted model_state.pt SHA-256:

4e40cca125c8fb4b866632708af5e0d5862cf66413bbbeb0a31c6a0385b30d65

The original release passed:

strict state-dict loading;
exact 339,788,800 parameter count;
tied embedding/head verification;
local tokenizer loading;
deterministic independent cold loads;
logit fingerprint matching.

The Hugging Face conversion additionally passed original-vs-standard-Llama logit parity and a fresh local AutoModelForCausalLM.from_pretrained() load.

Project status

This is the frozen 350M first-production release. Future CascaMini work may use larger or more diverse data corpora, but those are separate experimental lineages and do not change this release.