GPT-1900 D34 Physics CLM (Pre-1900 Only)
3.29B parameter GPT-1900 with continued pretraining on pre-1900-only physics texts (anachronism-filtered).
Trained from the pre-1900-only base model (not the standard d34-22btok which includes some post-1900 data).
This is an intermediate checkpoint in the pre-1900-only pipeline:
pre-1900-only base → physics CLM (pre-1900) → v3 SFT → discovery RL.
3 checkpoints saved at steps 4800, 9600, 14399 (5 epochs over pre-1900 clean physics data).
Model Details
- Architecture: Custom GPT with RoPE, QK-norm, ReLU², value embeddings (ResFormer)
- Parameters: 3.29B
- Layers: 34, Hidden dim: 2176, Heads: 17
- Context length: 2048 tokens
- Training data:
physics_clm_data_pre1900_clean
- Base model:
d34_pre1900 (pre-1900 only base)
