Continued Pretraining • 50M Parameters • 5K Context
Supra-1.5-Base-EXP
Supra-1.5-50M-Base-exp is a continued-pretrained 50M parameter Llama-style base model derived from SupraLabs/Supra-50M-Base. The context window has been expanded from 1,024 to 5,120 tokens using RoPE scaling and full-weight continued pretraining.
This specific repository has been modified to natively support ChatML. The tokenizer and embedding layers have been updated to include <|im_start|> and <|im_end|> as single special tokens.
Architecture & Updates
Specification
Value
Architecture
LlamaForCausalLM
Parameters
~50M
Vocabulary Size
32,002 (Expanded for ChatML)
Hidden Size
512
Layers
12
Attention Heads
8
KV Heads
4
Context Length
5,120 tokens
Tokenizer
BPE tokenizer (ChatML Jinja template applied)
Native EOS Token
`<
Vocabulary Expansion & Initialization
The model's 3B token CPT corpus included raw ChatML text. As a result, the base model originally learned the tags <|im_start|> and <|im_end|> as sequences of 7 separate subwords.
To convert these tags into single tokens without destroying the learned pre-trained representations, we used subword-mean initialization:
Extracted the attention embeddings for the original 7-token sequences.
Averaged the weights for each sequence.
Assigned these mean weights to the new single-token IDs.
This ensures stable loss at the start of fine-tuning and allows inference frameworks to natively stop generation when <|im_end|> is predicted.
Continued Pretraining Objective
This is a base model, not an instruct model. Training used packed raw text with standard causal language-modeling loss:
labels = input_ids
all non-pad tokens are trained
no response-only masking
no system/user/assistant masking
no LoRA adapters in the default run
Data Mix
The training mix prepared for the CPT run was:
3,000,000,062 CPT tokens
30% Tool Calling
30% ChatML Conversations
25% Factual Text (articles, essays, blogs)
15% Math & Logic Questions
Fine-Tuning Guide
This model is intended for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), including reasoning-style (R1) alignments.
You can use standard Hugging Face tools and call tokenizer.apply_chat_template() directly on your datasets.
LoRA Configuration Note:
Because the vocabulary size was expanded from 32,000 to 32,002, you must train the embedding layers during fine-tuning so the model can update the new ChatML tokens. Add the following to your PEFT/LoRA configuration:
