Chess Transformer 437M — FT3h (best Elo)
Greedy policy checkpoint from the exp191 meta-attention line. Best measured Elo in this project: ~1950 (Stockfish LimitStrength gauntlet).
| File | Description |
|---|
best_model.pt | Inference weights (~1.75 GB). Keys: model_state_dict, config, vocab_version, training meta. |
config.json | Architecture + FT3h training hyperparameters |
elo_eval.json | Full greedy gauntlet dump |
README.md | This card |
Play checkpoint: use greedy argmax over legal moves (temperature 0). Weak MCTS settings previously hurt Elo vs greedy.
Architecture
- 437M params (
hidden_dim=1280, num_layers=18, num_heads=20)
- Strengthened board encoder + meta-factored attention + Shaw-on-pos
- Compact move vocab (1968), STM-normalized fused board tokens
- Policy + 3-class value head
- Trained with NorMuon + PolarExpress,
torch.compile, cautious WD
Elo (greedy, movetime 0.05s SF side)
| SF Elo | Score | W-D-L (16 games) |
|---|
| 1450 | 0.750 | 10-4-2 |
| 1600 | 0.562 | 6-6-4 |
| 1750 | 0.562 | 7-4-5 |
| 1900 | 0.594 | 6-7-3 |
| 2050 | 0.312 | 2-6-8 |
Estimate ~1950 (bracketed by SF1900 / SF2050). Black is weaker at high SF; French / English / Réti openings are relative weak spots. Losses are mostly tactical checkmates.
Training lineage
1exp191 pretrain / early soft FTs
2 → exp191_soft_ft3_lichess
3 → exp191_soft_ft3b_unseen (~1832 Elo)
4 → exp191_soft_ft3h_edge_end ★ (~1950 Elo) ← this upload
Later soft / loss / tactics corrective FTs (ft3e/f/g/i, loss_fix, blunder_hard, tactics_hard) regressed Elo and were discarded. Do not prefer them over this file.
FT3h recipe (this checkpoint)
| Setting | Value |
|---|
| Init | exp191_soft_ft3b_unseen/best.pt |
| Steps | 6000 (best @ step 5000 by top1) |
| Batch | 256 |
| Soft frac | 0.85 |
| Soft α (vs hard CE) | 0.38 |
| Deep mix | 0.42 |
| Muon / Adam LR | 0.0009 / 4.5e-5 |
| Select metric | top1 (not soft_loss) |
| HFlip | 0.5 |
Data mixture used at FT3h
1. Shallow soft MultiPV (~2.0M train) — outputs/exp186_sf_multipv/soft_cache.pt
Stockfish MultiPV=8 soft targets on general positions (exp186 harvest from Lichess/SF-labeled position corpora such as avewright/chess-positions-lichess-sf / avewright/chess-positions-sf-labeled). Also mirrored on HF as dataset work under avewright/chess-soft-multipv-lichess.
2. Deep soft mix (~3.24M train) — outputs/unseen_quality_mix_ft3h/soft_cache.pt
Built to avoid overlap with prior FT caches (~33.9M excluded keys):
| Component | Role |
|---|
Pool: soft_cache_virgin_b_12m.pt (Lichess cloud-eval soft) | Unseen deep positions, phase-balanced → ~3.1M |
| Phase / edge / endgame / puzzle / syzygy spice | Attempted; most spice was already in the exclude set so little new volume landed |
| Mean label depth (mix report) | ~71 (cloud-eval style depths; quality varies by source) |
| Phase mix | ~21% opening / 46% middlegame / 33% endgame |
3. Hard ballast (~1.0M / step mix) — hard_ballast_d15_n2000000_s42.pt
Hard CE on Stockfish best-move labels, min depth ≥15, for stability when soft frac < 1.
Prior stage data (inherited via FT3b init)
FT3b was fine-tuned from ft3_lichess on:
- Same shallow
exp186 soft cache
- Deep:
soft_cache_unseen_ft3.pt (~3.2M Lichess-eval soft, held out from earlier FT3)
- Soft α 0.55, deep mix 0.55, hard ballast
Weights therefore already encode earlier Lichess MultiPV soft pretraining before the FT3h edge/end-lean mix.
Related public datasets
avewright/chess-soft-multipv-lichess — soft MultiPV cachesavewright/chess-soft-syzygy — syzygy soft (~498k)Lichess/chess-puzzles — puzzles (used in later experiments; not a major FT3h spice contributor)
Load
1import torch
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download("avewright/chess-transformer-437m-ft3h", "best_model.pt")
5ckpt = torch.load(path, map_location="cpu", weights_only=False)
6state = {k.replace("_orig_mod.", ""): v for k, v in ckpt["model_state_dict"].items()}
7config = ckpt["config"] # dict → ChessTransformerConfig(**config)
8# build_model(config); load_state_dict(state); vocab_version = ckpt.get("vocab_version", "compact")
Requires this repo’s ChessTransformer / compact move vocab (MOVE_VOCAB_VERSION=compact).
License
Weights: MIT (same as project). Upstream Lichess eval / puzzle data retain their respective terms (puzzles: CC0).
