Decoder-only language model trained from scratch on FineWeb (sample-10BT subset). GPT-2 scale (~123.7M parameters), pre-norm, SDPA attention, weight-tied embeddings.

Part of the transformer-room project — public training artifact for Phase 1 / Stage 1 baseline.

Model Architecture

Property	Value
Model type	`baseline_decoder` (custom, not a `transformers.PreTrainedModel`)
Parameters	~123.7M
Hidden size	768
Attention heads	12
Layers	12
Max sequence length	1024
Vocabulary size	50258 (GPT-2 base + `<PAD>` token)
Norm placement	`pre` (pre-norm — no LR warmup required)
Attention	Scaled Dot-Product Attention (PyTorch SDPA)
Weight tying	`True` (input embedding = output projection)
Positional encoding	Sinusoidal
FFN activation	ReLU

Training Details

Property	Value
Dataset	HuggingFaceFW/fineweb (`sample-10BT`)
Validation set	`CC-MAIN-2024-10` (held-out FineWeb CC dump, no train overlap)
Token budget	~2.5B (Chinchilla-optimal for 124M: 20 tokens/param)
Optimizer	AdamW (`lr=0.0006`, `weight_decay=0.1`)
LR schedule	Cosine decay to 0.1× (no warmup — pre-norm stable)
Effective batch size	512 seqs × 1024 tokens = 524,288 tokens/step
Micro batch / accum	64 seqs / 8 gradient accumulation steps
Hardware	H100 80GB
Precision	bf16 autocast (PyTorch AMP)
torch.compile	Yes (default mode, activation memory budget 0.75)

Results

Metric	Smoke run (1k steps)	Full run (~2.5B tokens)
Train loss (final step)	N/A	TBD
Train bits-per-byte	N/A	TBD
Val loss (`fineweb-cc-2024-10`)	N/A	TBD
Val bits-per-byte	N/A	TBD
HellaSwag (0-shot)	TBD	TBD
ARC-Easy (0-shot)	TBD	TBD

HellaSwag / ARC evaluations via lm-evaluation-harness — planned in ASH-21.

W&B Training Logs

Run	Link
Smoke run (1k steps, B=512)	N/A
Full run (~2.5B tokens)	TBD — training in progress

Loading the Weights

This model is not a transformers.PreTrainedModel and cannot be loaded with AutoModelForCausalLM.from_pretrained().

python

from safetensors.torch import load_file
from transformers import AutoTokenizer

# Tokenizer (GPT-2 base + <PAD> token, vocab_size=50258)
tokenizer = AutoTokenizer.from_pretrained("Ashwin7/gpt2-124m-fineweb-baseline")

# Weights
state_dict = load_file("model.safetensors")  # or hf_hub_download("Ashwin7/gpt2-124m-fineweb-baseline", "model.safetensors")

To reconstruct the model, instantiate BaselineModel from transformer-room/src/components/models/baseline_model.py using the fields in config.json, then call model.load_state_dict(state_dict).

License

Apache 2.0