AdamW (beta 0.9/0.95, weight decay 0.1 excluding embeddings), WSD schedule with peak LR 5e-4 and 10% decay, 2000 warmup steps. 19,073 steps at 524,288 tokens per step. ~18 hours on a single H100 SXM at ~35% MFU.

Final training loss: 2.507 (perplexity ~12.3)

Results

Benchmark	Score
HellaSwag (acc_norm)	30.5
PIQA	55.5
ARC-easy (acc_norm)	37.7

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "strectelite/PebbleGPT-320M", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("strectelite/PebbleGPT-320M")

Note: no KV cache implemented yet, so generation is slow.