This model is intentionally untrained. It is meant to test code paths, dataset packing, loss plumbing, checkpointing, evaluation loops, and distributed training setup without downloading or training a large model.

Specs

Architecture: ModernBertDecoderForCausalLM
Parameters: ~5M
Layers: 6
Hidden size: 256
Attention heads: 4
MLP intermediate size: 384
Context length: 512
Attention: sliding-window attention in every layer
Tokenizer: 4096-token tokenizer reused from SupraLabs/Supra-Mini-v4-2M
Weights: random initialization

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "sileod/microlm-ettin-swa-5m"

tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)