Architecture: LlamaForCausalLM
Base model size: 1.7B
Precision on disk: bfloat16
Tokenizer: extended SmolLM2 tokenizer with 36 additional special tokens (<assistant> + 35 <charter_X.Y> tokens), vocab size 49280

EPE Pretraining

This model was pretrained with on-the-fly reflection insertion using the reflection_1p column from the annotated sidecar dataset. The training augments standard autoregressive NTP with:

Reflection insertion: reflections are inserted into annotated documents at inference time; the model predicts the reflection tokens using CE loss
Constitution predictor: a multi-label BCE loss at the <assistant> token position trains the model to predict 35 charter items
Attention masking: post-reflection context tokens are blocked from attending to the reflection region
Position aliasing: post-reflection context tokens alias back to the same RoPE positions as pre-reflection context, making inference (without reflections) positionally equivalent to training

Chat Templates

Two named chat templates are provided:

Name	Use case
`default`	Standard SFT — plain `assistant` role token
`epe`	Activates constitution head — uses `<assistant>` (token 49152) at start of assistant turns

python

tok.apply_chat_template(messages, chat_template="default")  # standard
tok.apply_chat_template(messages, chat_template="epe")      # constitution head active

SFT Notes

Always use the bundled tokenizer (vocab size 49280); the original SmolLM2 tokenizer (49152 tokens) will mismatch embeddings for IDs 49152–49279
vocab_size=49280 is set in config.json

Raghav-Singhal/epe-1p-smollm-1p7b-100B-20n-2048sl-960gbsz

epe-1p-smollm-1p7b-100B-20n-2048sl-960gbsz

Details

EPE Pretraining

Chat Templates

SFT Notes

Raghav-Singhal/
epe-1p-smollm-1p7b-100B-20n-2048sl-960gbsz