MXFP4 container for colibrì, a Rust MoE inference engine for single-box streaming-expert inference on a DGX Spark (GB10).

This repository exists so the model can be downloaded and run directly, with no conversion step. The routed experts are QAT-native MXFP4 and are copied through bit-exact — a dequant→requantize round trip through another 4-bit format measured 6.40% rel-RMS of pure loss and 5.9% more bytes, i.e. worse on both axes.

43 layers, every one MoE — 256 routed experts top-6 plus 1 shared.

Context runs to the model's full 1,048,576 tokens, at 13.4 KB/token of KV.

An earlier revision of this card reported a ~40k ceiling at 206.9 KB/token. That was an engine limitation, not the checkpoint's — the KV cache allocated the full context for the raw latent tier even though only the 128-token window is ever read — and it has since been fixed: the raw tier is a ring sized to the window, so those rows are a per-sequence cost rather than a per-token one. V4 is now cheaper per token than a 120B hybrid (13.4 vs 16 KB) despite caching KV on all 43 layers.

decode is the noisiest column here. The eight reps behind that median spanned 4.53-4.78 tok/s forward-only in one session — a 5.5% spread with no code change. Treat a single suite as one draw, and do not read a difference against an earlier revision of this card as a speedup.

Method, caveats and the full fleet table: SpeedyColibri README.

The engine is github.com/GriffinPilz/SpeedyColibri. With it cloned and built, one command downloads this container and one serves it:

serve.sh fetches the container itself when it is missing, so the first line is optional. To download it by hand instead, mind the directory name — the registry resolves deepseek-v4-flash to -container-v2, and any other path will serve nothing:

DeepSeek for the model and its reference implementation; unsloth for the fp8 checkpoint this was converted from; colibrì for the original streaming-expert idea this engine grew from.