SNIP means Small Neural Intelligence Prototype. SNIP-0.4M is a deliberately
tiny causal transformer designed to make the complete language-model lifecycle
inspectable on ordinary hardware.
Architecture
approximately 400,000 parameters;
3 transformer layers;
96 hidden dimensions;
3 attention heads;
128-token context;
custom 512-token byte-level BPE tokenizer;
GPT-2-compatible architecture saved with Hugging Face Transformers.
Training
The base model is initialized from scratch and pretrained on a bounded stream from
roneneldan/TinyStories. It does not inherit pretrained weights.
powershell
1uv run python projects/snip-0.4m/prepare_data.py
2uv run python projects/snip-0.4m/train.py
3uv run python projects/snip-0.4m/evaluate.py
Intended use
SNIP is an educational and experimental base model. Its small capacity makes it useful
for testing tokenizers, adapters, curricula, evaluation pipelines, compression, and
RL methods quickly. It is not intended as a factual assistant or production chatbot.
Results
Measured parameter count, held-out loss, perplexity, throughput, and generation samples
are written to artifacts/snip-0.4m-base/evaluation.json after training.
Base v1
Parameters: 397,152
Pretraining tokens: 1,638,400
Held-out loss: 3.3656
Held-out perplexity: 28.9508
These are local CPU results over 200 held-out 128-token blocks.
SNIP-0.09M compression variant
The compressed student has 87,360 parameters: two layers, 48 hidden dimensions,
three attention heads, and the same 512-token vocabulary and 128-token context. It
was trained for 500 steps over 1,024,000 tokens.
Variant
Parameters
Held-out loss
Perplexity
SNIP-0.4M teacher
397,152
3.3656
28.95
SNIP-0.09M scratch
87,360
3.8325
46.18
SNIP-0.09M distilled
87,360
3.7831
43.95
The distilled student is 4.55 times smaller than its teacher and improves perplexity
by 4.82% over the architecture-matched scratch run. It still trails the larger
teacher, making the compression-quality tradeoff explicit.
powershell
1uv run python projects/snip-0.4m/train_distilled.py --steps 500
2uv run python projects/snip-0.4m/train_distilled.py --steps 500 `
3--distillation-weight 0 --output-name snip-0.09m-scratch
Story LoRA
The first post-trained variant attaches a rank-4 LoRA adapter to the attention and
feed-forward projections. It learns an instruction-shaped microfiction format while
leaving the 397K base weights frozen.
powershell
1uv run python projects/snip-0.4m/train_lora.py
2uv run python projects/snip-0.4m/evaluate_variants.py
The run saves both a lightweight adapter and a standalone merged variant.
Pairwise reward model
SNIP Reward reuses the pretrained 397K transformer and replaces the language-model
head with a scalar ranking head. It learns from 810 structured preference pairs and
is evaluated on 90 held-out pairs.
uv run python projects/snip-0.4m/train_reward.py --epochs 4 --batch-size 128
The 397,248-parameter model reached 100% pairwise accuracy on both wrong-goal and
wrong-lesson corruptions, with a mean chosen-minus-rejected margin of 3.3656.
This is a narrow synthetic consistency test, not evidence of general human-preference
alignment.
Verifiable-reward language RL
SNIP-GRPO applies an 18,432-parameter LoRA policy to parity, comparison, and threshold
prompts with exact binary rewards. Each update samples 16 actions per prompt, centers
and normalizes reward inside each group, clips the policy ratio, and penalizes KL
divergence from the frozen base policy.
uv run python projects/snip-0.4m/train_grpo.py --steps 300
Across 176 held-out prompts, greedy accuracy improved from 52.27% to 69.32%.
Less-than accuracy rose from 47.50% to 77.50%, and sum-threshold accuracy rose from
57.89% to 65.79%. Parity stayed at 50%, an explicit failure mode rather than a hidden
average. The method is GRPO-style group-relative policy optimization over a constrained
two-token action space; it is not a full open-ended reasoning model.
Logic SFT control
An architecture- and data-matched supervised LoRA control trained for 300 updates on
the same 684 examples and evaluated on the same 176 held-out prompts:
Variant
Overall
Less-than
Parity
Sum threshold
Frozen base
52.27%
47.50%
50.00%
57.89%
Logic SFT
69.32%
80.00%
15.00%
72.37%
GRPO
69.32%
77.50%
50.00%
65.79%
SFT and GRPO tied in aggregate but did not learn the same behavior. SFT was stronger on
comparison and thresholds while catastrophically degrading parity; GRPO preserved
base parity performance. The merged SFT artifact was independently reloaded and
re-evaluated after training.
