Pure-NumPy reference implementation of the DeepSeek-V4-Tiny architecture (MLA + Hyper-Connections + MoE).
Maintained by eulogik — building DIME, the disk-streaming Mixture Engine for running 284B-class MoE models on 16 GB Macs.
No PyTorch. No GPU. No CUDA. Just numpy + safetensors. And it matches the
official PyTorch reference exactly — cosine similarity = 1.000 on logits,
verified across 1–9 tokens.
This is the reference engine behind DIME
(Disk-streaming Mixture Engine) — the runtime that runs 284B-class MoE models on
a 16 GB Mac Mini by streaming experts from disk.
⚡ Run a 110M-parameter MoE model in ~10 lines, in the browser, on a phone, or
on a toaster. The weights are ~441 MB and load with zero ML frameworks.
nanowhale-100m is a 110M-parameter DeepSeek-V4-Tiny that distills the key
ideas of DeepSeek-V4 into a tiny, hackable package:
Component
Detail
Layers
8
Attention
Multi-head Latent Attention (MLA) with RoPE
Experts
4 routed + 1 shared, top-2 routing
Hyper-Connections
hc_mult=4 with Sinkhorn-iterating mixer
MTP
Multi-Token Prediction head
Params
~110M (float32, ~441 MB safetensors)
All of it is reimplemented in readable NumPy in a single file
(nanowhale_dime.py) that you can read end-to-end in one
sitting. Great for learning how modern MoE + MLA models actually work under the
hood.
✅ Why "bit-exact"?
We compared our NumPy forward pass against the official PyTorch reference model
on logits for sequences of length 1, 4, and 9 tokens:
MLA query/key de-rotation (inverse RoPE on attention output)
The Softplus + gate-bias router normalization
Hyper-Connection pre/post mixing with Sinkhorn-balanced combination matrix
Shared + routed expert summation with top-2 scaling
🌊 What is DIME?
DIME is an MoE inference runtime for Apple Silicon that breaks the "must fit in
RAM" rule. It lazily loads expert weights from local or networked NVMe and keeps
a frequency-promoted hot tier of the experts you actually use (the same
trick ktransformers uses to run 236B models on consumer hardware).
nanowhale-100m-DIME is the fully self-contained, verified reference. Once the
math is proven at 110M params, the same engine scales to
DeepSeek-V4-Flash-class (284B) models — streamed, not loaded.
Follow eulogik for the 284B-on-a-Mac-Mini
release. 🍎
📎 Original model
Weights and tokenizer are from cmpatino/nanowhale-100m.
This repo contains only the NumPy inference engine + Colab + this card.
We build DIME (Disk-streaming Mixture Engine) — MoE inference for Apple Silicon
that runs models far larger than RAM by streaming experts from disk. Follow
@eulogik and star
eulogik/dime for the 284B-on-a-Mac-Mini release. 🍎
📄 License
Apache-2.0 (weights inherit their original license).
