zero-125m is a 124,680,960-parameter OLMo2-compatible base language model trained from scratch on 2,500,001,792 FineWeb-Edu tokens. It is the released dense model from the lm-from-zero project, which covers the complete path from byte-level BPE tokenization to local serving.

This repository includes the native Transformers model and tokenizer, F16 and Q8_0 GGUF derivatives, evaluation evidence, and the full technical report.

Local base-model continuation in Pi using the Q8_0 GGUF through llama.cpp

Technical report | Source and reproducibility | Capture evidence

Intended use

The model is intended for research, education, systems experiments, conversion tests, and local inference demonstrations. It is a pretrained base model. It has not been aligned for assistant use and should not be treated as a reliable chatbot, factual reference, or safety-critical system.

Training

Architecture: OLMo2-compatible causal decoder
Parameters: 124,680,960
Layers: 12
Hidden width: 768
Query heads: 12
Key/value heads: 4
Context length: 2,048 tokens
Vocabulary: 32,000 byte-level BPE tokens
Training tokens: 2,500,001,792
Data: FineWeb-Edu sample-10BT, immutable revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9
Optimizer: AdamW with linear warmup and cosine decay
Numerical format: bfloat16 training with fp32 optimizer state
Hardware: one NVIDIA RTX 4080 SUPER

The tokenizer, shard manifest, model configuration, checkpoint, evaluation, and converted package are connected by recorded hashes. The native Transformers export has an exact maximum fp32 logit error of 0.0 against the project checkpoint in the recorded parity evaluation.

model.safetensors stores float32 tensors and is 498,738,888 bytes. That precision is deliberate: it is what makes the exact logit-parity check above meaningful, because the released file is bit-identical in value to the project checkpoint rather than a re-quantized copy of it. Load with dtype=torch.bfloat16 for inference, or take one of the GGUF files below if a smaller download matters more than parity.

Evaluation

Evaluation uses fixed, non-wrapping FineWeb-Edu windows.

Split	Mean loss	Perplexity
Validation	3.324985	27.799
Test	3.400198	29.970

These measurements describe next-token likelihood under this tokenizer and window protocol. They are not instruction-following, factuality, or safety scores.

Training loss against tokens consumed for the 20M and 125M dense models

Validation and test perplexity for the 20M and 125M dense models

The figures/ directory also carries the 20M architecture study, the dense ablation screening, and the local judge calibration from the wider project.

Transformers usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tarpous/zero-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

This is a base model, so continuations may be repetitive, abrupt, or inaccurate. A prompt formatted as a conversation does not turn it into an instruction-following model.

GGUF files

File	Size	SHA-256
`zero-125m-f16.gguf`	250,567,744 bytes	`4863f80ea315fc0713da86750f71cddc8dfbec09c0d3f8a853cee650220027ab`
`zero-125m-q8_0.gguf`	133,708,864 bytes	`401866c0a19e803eea19b00e056915a37b05ebe1146430b618355071680b8692`

The Q8_0 artifact produced 76.5 decoded tokens per second in the recorded bounded CUDA llama.cpp smoke. Throughput is hardware and build dependent. Conversion details and the pre-tokenizer compatibility patch are documented in the project repository.

Broader project evidence

The project also trained a 20M dense FineWeb model and separate TinyStories models near 20M parameters using dense attention, Mamba-2, and masked discrete diffusion. It includes dense ablations, supervised fine-tuning, DPO, and local judge calibration. Those results describe the research project and are not additional training stages of this released base checkpoint.

The full report is available in this model repository as lm-from-zero-technical-report.pdf and in the GitHub release.

Limitations

The model is small and was trained for only 2.5 billion tokens.
It is a base model without safety or instruction tuning.
FineWeb-Edu may contain errors, bias, personal information, and other undesirable web content despite dataset-level filtering.
The model can generate false, biased, offensive, repetitive, or incoherent text.
Evaluation covers held-out language-model loss, not downstream capability or responsible-deployment criteria.
The planned 125M FineWeb Mamba-2 and masked-diffusion runs were not completed.
The planned 500-prompt cross-judge calibration was not completed.

Users are responsible for evaluating the model in their own setting and for adding safeguards appropriate to any application.

Citation

bibtex

@software{tarpous2026lmfromzero,
  author = {tarpous},
  title = {lm-from-zero: A Local-First, Reproducible Language-Model Pipeline from Tokenization to Serving},
  year = {2026},
  url = {https://github.com/tarpous/lm-from-zero}
}

License

The model weights, tokenizer, GGUF files, and code are released under Apache-2.0. The technical report and its figures are released under CC BY 4.0.