Name: GPT-X2.5-135M
Availability: InStock

You can deploy this model and start earning money today!

Views

No views yet

GPT-X2.5-135M is an improved 2.5th generation model in the GPT-X family built on the TX-3 architecture. 135M parameters, 75B tokens, custom 32K tokenizer, 30 layers, and our XGQA attention. Trained from scratch on a multi-source curriculum achieving near state-of-the-art performance on both natural language and mathematical reasoning benchmarks.

Results

GPT-X2.5-135M achieves competitive performance with leading models despite using significantly less training data. It reaches a 25.17 Intelligence Index—within 1.96 points of SmolLM2-135M while using about 27× fewer training tokens.

Benchmark results and Intelligence Index methodology follow the Open SLM Leaderboard. Training compute is estimated as 6 × parameters × training tokens.

Rank	Organization	Model	Parameters	HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3	Int Index	Training tokens
1	HuggingFace	SmolLM2-135M	135M	43.22%	58.63%	29.69%	68.44%	39.20%	27.13	2T
2	HuggingFace	SmolLM-135M	135M	42.70%	56.31%	29.01%	68.28%	36.80%	25.74	600B
3	Axiomic Labs	GPT-X2.5-135M	135M	40.57%	51.81%	29.18%	69.42%	38.40%	25.17	75B
4	Facebook	MobileLLM-R1-140M-base	140M	33.84%	49.92%	24.74%	63.22%	65.70%	24.64	4.2T
5	Axiomic Labs	GPT-X2-125M	125M	40.41%	51.47%	27.82%	67.30%	37.20%	23.36	75B
6	BananaMind	BananaMind-2-Pro-Preview	139M	39.83%	51.01%	27.13%	66.76%	38.90%	23.04	52B
7	Axiomic Labs	GPT-X-125M	125M	36.57%	50.76%	26.62%	64.96%	35.60%	19.94	15B
8	SupraLabs	Supra2-100M-Base	100.7M	35.98%	47.81%	24.83%	65.40%	36.90%	19.41	30B
9	Facebook	OPT-125M	125M	31.31%	40.28%	22.70%	62.24%	35.20%	13.80	180B
10	OpenAI	GPT-2 (124M)	124M	31.26%	39.35%	22.35%	62.08%	35.70%	13.58	10B

The Intelligence Index chance-normalizes HellaSwag, combined ARC, PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.

The model also achieves an overall elo of 1104 on the Bananamind-Base-Bench-1.1 and can be viewed on the BananaMindBench Leaderboard.

What's New in 2.5?

Change	GPT-X2	GPT-X2.5	Why
Parameters	125.08M	135.03M	More capacity is concentrated in the feed-forward layers
Feed-forward width	1,536	1,728	Expands each SwiGLU MLP from 2.67x to 3x the 576-wide residual stream
Context length	1,024	8,192	Training starts at 2K context and extends to 8K after 60B tokens
Attention refinement	QK-Norm	XSA projection; no QK-Norm	Projects out the component of each attention-head output parallel to its current-token value vector
Embedding scaling	Multiply by sqrt(576)	None	Matches the formulation used for the XSA training run
Tokenizer	32,768-token custom BPE	32,770 tokens (32K BPE + ChatML start/end)	Adds native chat-boundary tokens without replacing the base tokenizer
Training data	75B tokens, 4-source curriculum	75B tokens, expanded 11-source curriculum	Adds Cosmopedia, SmolTalk, and four FinePhrase subsets to the original mix

Architecture

Component	Details
Position encoding	RoPE (theta=100,000)
Normalization	RMSNorm (float32 upcast)
Feed-forward	SwiGLU (3-matrix gated MLP, 1,728 intermediate width, 3x ratio)
Attention	XGQA -- 9Q / 3KV (3:1)
Attention refinement	XSA projection; no QK-Norm
Bias	None (all layers bias-free)
Embedding	Weight tying, without embedding scaling
Depth	30 layers x 576 hidden
Maximum context	Trained to 8,192 tokens

Config

vocab_size     = 32,770    (32K custom BPE + ChatML start/end tokens)
n_layer        = 30
n_head         = 9         (query heads)
n_kv_heads     = 3         (key-value heads, 3:1 GQA)
n_embd         = 576
head_dim       = 64
intermediate   = 1,728     (SwiGLU, 3x ratio)
block_size     = 8,192
rope_theta     = 100,000
xsa_projection = true
qk_norm        = false
embedding_scale = false
total params   = 135,032,256

Parameter Breakdown

Component	Params
Token embeddings (32770 x 576)	18,875,520
Per block (x30): attention + SwiGLU + norms	3,871,872
30 transformer blocks	116,156,160
Final RMSNorm	576
LM head (tied with embeddings)	0
Total	135,032,256

Training

Data

Eleven sources are mixed per optimizer step with a progressive curriculum. The run begins with mostly educational and general web text, ramps math and code, introduces synthetic educational and conversational data, then switches to a FinePhrase-heavy final stage at 58B tokens.

Source	Dataset	Purpose
FineWeb-Edu	HuggingFaceFW/fineweb-edu (`sample-100BT`)	Primary educational web text
DCLM	mlfoundations/dclm-baseline-1.0	High-quality diverse web text
FineMath	HuggingFaceTB/finemath (`finemath-4plus`)	Mathematical reasoning (score >= 4)
NPset-2-Python-Edu	AxiomicLabs/NPset-2-Python-Edu	AST-normalized Python code
Cosmopedia	HuggingFaceTB/cosmopedia	Synthetic textbooks, stories, and educational content
ArithMark-2	AxiomicLabs/ArithMark-2.0	Focused arithmetic practice
SmolTalk	HuggingFaceTB/smol-smoltalk	Chat-formatted instruction and conversation data
FinePhrase	HuggingFaceFW/finephrase (FAQ, math, table, and tutorial subsets)	Structured explanatory and knowledge-dense text

Tokens: 75B
Tokenizer: 32,770-token custom BPE (32K base vocabulary plus ChatML start/end tokens)
Validation: held-out FineWeb-Edu; best recorded validation loss 2.671957

Progressive Data Curriculum

The configured weights change at discrete token thresholds. All percentages below come from the effective (second) CURRICULUM definition in the training script:

Token range	FineWeb-Edu	DCLM	FineMath	Code	Cosmopedia	ArithMark-2	SmolTalk	FinePhrase
0 -- 18B	58%	40%	1%	1%	0%	0%	0%	0%
18 -- 18.5B	56%	39%	3%	2%	0%	0%	0%	0%
18.5 -- 19B	54.5%	37.5%	5%	3%	0%	0%	0%	0%
19 -- 20B	54%	37.5%	5%	3.5%	0%	0%	0%	0%
20 -- 40B	54%	36%	6%	4%	0%	0%	0%	0%
40 -- 45B	52%	36%	6%	4%	2%	0%	0%	0%
45 -- 48B	50.8%	36%	6%	3.7%	3%	0.5%	0%	0%
48 -- 50B	49.5%	36%	6%	3.5%	4.5%	0.5%	0%	0%
50 -- 53B	48.5%	35%	6%	2.5%	7%	1%	0%	0%
53 -- 55B	48%	35%	6%	1.5%	8.5%	1%	0%	0%
55 -- 58B	46%	32%	6%	1%	9%	1%	5%	0%
58 -- 75B	0%	33%	11.5%	0%	0%	1%	5%	49.5%

The final FinePhrase share is split into FAQ (14.85%), math (14.85%), table (7.425%), and tutorial (12.375%). The loader preserves the configured proportions across each optimizer step and shuffles source order; if a source is unavailable, its share is redistributed proportionally among the active sources.

Optimization

Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.01)
Learning rate: 1.5e-3 max, decays to 0
Schedule: WSD -- 2,000-step warmup, stable through 90% of training, then linear decay to 0 over the final 10%
Batch size: 524,288 tokens (micro_batch=4-1, seq_len=2048-8192, grad accum)
Precision: bfloat16 mixed precision
Gradient clipping: 1.0

Hardware

1x RTX 3080 Ti
Training time: ~800 hours

Citation

bibtex

@misc{gptx2_2025,
  title={GPT-X2.5: Data-Efficient Language Modeling at 135M Scale},
  author={Axiomic Labs},
  year={2026},
  howpublished={\url{https://huggingface.co/AxiomicLabs/GPT-X2.5-135M}},
  note={Trained on 75B tokens with a progressive curriculum and custom tokenizer}
}