Architecture: Frankenstein LLM combining best practices
- BitNet b1.58 (Microsoft) - 1.58-bit ternary weights
- Llama-2 tokenizer (32k vocab)
- Llama 3 GQA (12Q/4KV heads for 3x KV cache reduction)
- Gemma 2/3 stability (QK-Norm + Logit Soft-Capping)
- Extended RoPE (theta=1M for long context)
Parameters: ~100M
Training: 500 steps on 765 samples (initial training)
Optimizer: Stanford Muon + AdamW hybrid

Architecture Specs

Component	Value
Hidden dim	768
Layers	12
Q Heads	12
KV Heads	4 (GQA)
Head dim	64
MLP dim	2048
Vocab size	32,000
Max seq len	2,048
RoPE theta	1,000,000

Training

Trained with the Muon optimizer from Stanford, which showed characteristic "Muon Drop" - steep loss reduction:

Initial loss: 10.59
Final loss: 1.04 (90% reduction in 6 minutes)
Hardware: RTX 2080 Super (8GB VRAM)

Files

distillix-v0.safetensors - SafeTensors format (382 MB)
distillix-v0.3.gguf - GGUF format for llama.cpp (191 MB)
model_500steps.pt - PyTorch checkpoint

Usage

python

import torch
from safetensors.torch import load_file

# Load model weights
state_dict = load_file("distillix-v0.safetensors")

# For inference, use with llama.cpp or bitnet.cpp
# GGUF file is provided for CPU inference

Limitations

Early training (500 steps) - model needs more training
Limited training data (765 samples)
Best used as a starting point for further fine-tuning

License

Apache 2.0

Citation

@misc{distillix2025,
  title={Distillix: Frankenstein BitNet b1.58 Knowledge Distillation},
  author={Seaburg, Riley},
  year={2025},
  url={https://github.com/rileyseaburg/distillix}
}

rileyseaburg/distillix-100m-v0.3

Distillix 100M v0.3

Model Details

Architecture Specs

Training

Files

Usage

Limitations

License

Citation

rileyseaburg/
distillix-100m-v0.3