Phase 1 (C4): 1× Chinchilla on allenai/c4, 9.65B tokens, peak LR 0.01, cosine decay to 5e-4. Final loss 2.97. Standalone checkpoint at mlnomad/goat-v-482m-c4-seed0.
Phase 2 (FineWeb-Edu): continue from the phase-1 checkpoint on HuggingFaceFW/fineweb-edu sample-100BT, ~4.7B more tokens, peak LR 0.002, cosine decay to 1e-4. The model + optimizer state is restored from phase 1; only the training corpus changes.

This is intended as a usable model, not an ablation: phase 2 reshapes the model toward the FineWeb-Edu distribution, which previous experiments showed is much closer to long-form benchmark distributions (PG19, etc.) than C4.

Architecture


Params	482.3M
Depth / n_embd / heads	d=22 / 1408 / 22
Attention	Yat kernel on RoPE'd residual head slices (no Q/K), V projection kept
MLP	YatNMN-Softplus (scalar bias + learnable α)
Value embeds	none
Total tokens	~14.4B (9.65B C4 + 4.7B FineWeb-Edu)
Hardware	TPU v6e-8 (europe-west4-a, TRC), fp32

Sister checkpoints

mlnomad/goat-v-482m-c4-seed0: phase-1 only (C4-only), final loss 2.97.
mlnomad/yatnmn-full-d22-chinchilla-pytorch: same architecture trained from scratch on FineWeb-Edu, 1× Chinchilla, final loss 2.57.
mlnomad/goat-v-482m-c4-seed0-flax-resume: raw flax/orbax checkpoint with optimizer state, used as phase-1 → phase-2 starting point.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/goat-v-482m-c4-then-fineweb-seed0",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

License

Apache 2.0.