A 1.0B-parameter ML/LLM-engineering specialist, pretrained from random
initialization on a single RTX 5090 — 16.0B tokens in ~13.5 days including
gates and recoveries — then one bounded knowledge/format SFT stage. Not a
general chatbot. By Home Brewed Labs (project codename MetaLLM).
The engineering core of the project is measurement discipline: every training
stage was gated by frozen batteries with exact checkpoint identity, replay
accounting (≈1.04B replayed tokens from two hardware incidents, fully
receipted), independent adversarial review of evaluation machinery, and a
sealed never-trained-on final holdout (H9) reserved for a single one-time
qualification.
Architecture
26-layer decoder, d_model 1792, GQA 28Q/14KV, SwiGLU (d_ff 4864), RMSNorm
pre-norm + QK-norm + z-loss, RoPE θ=500k with NoPE every 4th layer
(length-generalizing retrieval heads), document-masked attention, 40k
byte-level BPE with FIM sentinels, 2048 native context. Muon (2D) + AdamW,
WSD schedule, curriculum with short-sequence early phases.
trust_remote_code=True is required: this is a custom architecture
(QK-norm, NoPE interleave, document masking). Do not map to a stock
Llama config — that silently drops QK-norm/NoPE and produces a different
model than the one measured below. Stock GGUF/vLLM converters reject this
architecture by design; a faithful llama.cpp port is on the roadmap.
Evaluation (champion s2b075, frozen batteries, repeated where noted)
Capability
Metric
Metallum-1B
Qwen3-1.7B
Qwen3-0.6B
SmolLM2-1.7B
In-domain ML-eng problems
ineval_v1, frozen 150-task exec suite
7/150
7/150
3/150
0/150
In-domain code (internal suite)
code_v2 40-task, 2 exact repeats
32/40 (disclosure: known training overlap)
—
—
—
Long-context retrieval
RULER (90) / needles D1024+D2032 (18)
86/90 and 18/18
—
—
—
Base LM efficiency
held-out ML-arXiv BPC
0.7276
—
—
—
Final unseen generalization
H9 sealed holdout BPC (one-time run)
1.3953
—
—
—
All numbers were produced by frozen evaluators with sealed checkpoint
identity; the ineval suite was used for model selection during development
(selection-aware caveat), which is why the untouched H9 result is the number
that matters for release claims.
Structured output: by decoding, not by hope
Base weights score 0/60 on a strict whole-output format suite (JSON
schema / tool-call / MCQ-letter), and three bounded SFT recipes plus a
ReST-EM screen all failed to move that (all receipted). This model therefore
ships with decode-time structure enforcement: serve_metallum.py
(included) constrains generation with a token-level JSON grammar +
schema-forced keys, and scores 60/60 on the same frozen suite. If you
consume the raw weights without the wrapper, expect prose, not JSON.
ML/LLM-engineering assistant: PyTorch/training scaffolding, concept
explanation, long-context retrieval over technical docs, structured-output
endpoints via the wrapper. Out of scope: general chat, non-ML factual
QA, general-purpose coding, safety-critical use, autonomous code execution.
Honest limitations
neural_ops floor exception: this checkpoint scores 4/5 on the internal
neural-ops category vs the 5/5 preregistered floor. The exception was
adjudicated explicitly by the owner; it is disclosed, not hidden.
MCF (250q multiple-choice) is permanently non-promotable: the teacher
generation prompts embedded real MCF items, so no MCF number from this
lineage is valid evidence. Do not report one.
ineval is selection-aware: 7/150 was measured on a suite used during
development. The sealed H9 result (1.3953 BPC on 400k post-cutoff
Wikipedia characters, single run, contamination-verified at 0.0099% shingle
containment) is the generalization claim.
Free generation makes local factual slips; verify specifics.
No preference/safety alignment; no RLHF.
Format behavior requires the constrained decoder (above).
Training data
Pretraining: 19.399B effective tokens (8.473B unique): 53.6% code, 19.6%
knowledge, 15.2% reasoning, 11.6% math. 1.24% (240.6M tokens) is
provider-teacher-derived; full lineage map is maintained by the authors.
SFT (12,496 rows): 40% locally self-hosted open-teacher MCQs
(qwen2.5-coder:14b, Apache-2.0), 12% self-generated synthetic retrieval rows,
48% provider-teacher-derived replay (disposition: owner written-rights
determination, on file and available on request).
Attribution — Stack Exchange (stack_ml_edu, 2.973B pack tokens, 15.3% of
the pack): CC BY-SA 4.0. Attribution is owed to Stack Exchange contributors,
and the share-alike obligation attaches to that portion of the training data.
License
Apache-2.0 (model weights + code). See the attribution note above regarding
training-data share-alike obligations.
