ohara-moe-0.9B-a91M-base-d12
A
fine-grained mixture-of-experts pretrained base model: 911M total parameters,
91M active
per token (10x sparse). Trained from scratch with
ohara on 2xA100-80GB.
This is a base language model, not a chat model. It continues text; it
does not answer questions and never stops on its own. For something that
responds, use
ohara-moe-0.9B-a91M-chat-d12.
Use this one to run your own finetune from a sparse base.
Architecture
64 routed experts + 1 shared expert per layer, top-4 routing, on all 12 layers.
The shared expert is always active, so routed experts specialise instead of each
relearning the common transformation (DeepSeekMoE's shared-expert isolation).
| |
|---|
| Total parameters | 911M |
| Active per token | 91M |
| Sparsity | 10x |
| Experts | 64 routed + 1 shared, top-4 |
| Expert width | 448 (SwiGLU) |
| Layers / hidden / heads | 12 / 768 / 6 |
| Context | 2048 |
| Vocabulary | 50,304 |
Routing uses sigmoid gating with normalised weights (softmax over 64 logits
drives every weight tiny) and quantile balancing for load balance: the router
bias is solved in closed form each optimizer step, so there is no auxiliary loss
and no coefficient to tune. Experts dispatch through torch._grouped_mm -- one
kernel for all 64, no per-expert Python loop.
Results
Three architectures, identical tokens (1.48B), schedule, learning rates and seed:
| dense | MoE 8x top-2 | this model |
|---|
| Total / active params | 162M / 162M | 332M / 85M | 911M / 91M |
| Pretrain val bits/byte | 0.9062 | 0.8887 | 0.8652 |
| SFT val loss | 1.2084 | 1.1536 | 1.0824 |
| SFT val perplexity | 3.35 | 3.17 | 2.95 |
| SFT next-token accuracy | 69.40% | 70.40% | 71.73% |
4.5% better bits-per-byte than dense, and the gap widened through training
(2.2% at step 250, 4.1% at step 2000) rather than saturating. It reached the
dense model's final quality in about 29% fewer steps.
Honest caveats. Active FFN FLOPs are 1.094x the dense baseline (5 active
experts x 448 vs dense 2048), so roughly 1-2 points of that 4.5% is bought
compute rather than architecture. And 911M parameters is 5.6x the memory of the
dense model for 1.09x the compute -- MoE wins per token here, not per hour.
For scale: nanochat reaches GPT-2 grade at 0.718 bits/byte using roughly 28x more
compute than the 1.4e18 FLOPs spent here. This is a small model.
Files
model.safetensors, config.json, tokenizer. config.json is load-bearing:
moe_experts_per_tok and moe_gate_fn leave no trace in any tensor shape, so
without them the weights load correctly but route differently than they trained.
Usage
1import json
2from safetensors.torch import load_file
3from ohara.models.llama import Config, Llama
4
5cfg = json.load(open("config.json"))
6cfg.pop("architecture"); cfg.pop("iteration")
7model = Llama(Config(**cfg))
8model.load_state_dict(load_file("model.safetensors"), strict=False) # rotary rebuilds
