Built and validated in one night on a Dell laptop (RTX 4070 8GB, 31 GB RAM, Ubuntu 24.04), plus a GPU tier the following session: 50× total speedup, ~6 s/token and falling.
Abstract
We present dragon.c, a from-scratch C inference engine that runs Meituan LongCat-Flash-Thinking-FP8
(560B parameters, Mixture-of-Experts) on a consumer laptop with 31 GB of RAM. The engine reads the
model's native FP8 safetensors directly via mmap — no format conversion, no GGUF, no llama.cpp, no
Python in the hot path — and streams 531 GB of weights through 31 GB of memory using the OS page
cache as the eviction layer. Every mathematical operation was validated brick-by-brick against a
PyTorch reference to floating-point precision, culminating in a full 28-layer forward pass whose
output logits are IDENTICAL to the reference (same top-5 token IDs, logits within ~0.1). The engine
generates coherent English. Through four measured optimization rounds (f32 LRU cache → native-precision compute → AVX2+LUT
kernels → a GPU-resident attention tier on the RTX 4070) decode latency improved 50× to ~6 s/token,
with output invariant across every optimization. We document every dead-end as a control. The build demonstrates
the thesis this project exists to prove: the deficit in consumer-hardware AI is drivetrain, not
engine — organization of computation and data movement, not parameter count.
Shortcut-connected MoE: the MoE is computed on sub-block-0's normed hidden state but its output is added at the END of sub-block-1 — a designed overlap window
Precision
HYBRID: attention/norms/embeddings bf16, router f32, the ScMoE FFNs fp8 — both the routed experts AND the per-sub-block dense branches (e4m3, 128×128 blockwise f32 scales stored IN the shards, not in the index)
2. Method: validate every op, advance only on proof
The build discipline, applied without exception:
Dump ground-truth tensors from a working Python/PyTorch reference (ref/*.f32).
Write the C implementation of ONE operation.
Compare element-wise. Advance only when the diff is at floating-point-noise level.
Write the result (and every dead-end) into the permanent record before moving on.
This discipline caught real bugs at every stage and — critically — caught a corrupt reference
(§6) that would otherwise have burned days.
3. Phase results (every brick, with measured error)
Hand-deriving MLA from the paper failed (diff 1.7). Reading Meituan's modeling_longcat_flash.py
line-by-line revealed three undocumented-in-practice details, without which the attention is wrong:
mla_scale_q_lora = 2.0 — q_pass AND q_rot are multiplied by 2.0.
mla_scale_kv_lora = 3.4641016151377544 (= 2√3) — applied to the kv-lora vector AFTER its layernorm.
Interleave-before-rope (use_mla=True path): before rotary embedding, the rope dims are
reshaped [d/2,2]→transpose→[d] (even indices first, then odd). Softmax scale = 192^(-0.5) =
0.07216878. k_rot is computed ONCE per token and SHARED across all 64 heads.
With these three, hand-MLA matches the module to 6.9e-3 (bf16 noise). These constants are now
permanent record; they never have to be rediscovered.
Design (the "game-engine pattern" — assets in a PAK file, OS does the paging):
Manifest: Python pre-parses all 75 shard headers ONCE into manifest.bin — 43,756 fixed-width
records (name, shard id, dtype, absolute byte offset, shape, scale-tensor location). C mmaps it
and bsearches by name. No JSON parsing in C, ever.
mmap everything: all 75 shards are mmap'd read-only at startup. get_tensor returns pointers
into the maps. Weights are NEVER malloc'd/copied (in the final engine — see §8).
Eviction = the kernel's page cache. 531 GB streams through 31 GB because Linux pages weights
in on touch and evicts them under pressure. Zero eviction code. The OS is the streaming layer.
OpenMP across output rows of every matvec (14 threads).
Result: full 560B forward, 74 s cold. Layer-0 through the native read path matches Python to 2.2e-3.
6. The control that saved the build: the reference was poisoned
While validating the full layer, our composition differed from the saved reference by 3.48 — for an
hour this looked like a C bug. It was not. Root cause, found by checking weight.is_meta:
Meituan's router does self.classifier.weight (direct attribute access) instead of calling
self.classifier(x). Our Python-side lazy-loading hook fires on __call__ — so it NEVER fired
for the router. The router weight stayed on the meta device, the module's MoE routing was garbage,
and the garbage was silently baked into the saved reference tensor.
Force-loading the router and regenerating gave a clean reference — which our C matched to 1.9e-6.
The C was right all along; the reference was corrupt. Lessons, now permanent controls:
Never trust a module-derived reference until 0 meta params is verified.
Any op that touches .weight directly bypasses load hooks.
dragon.c loads every weight explicitly and is structurally immune to this bug class — the
strongest single argument for owning the engine instead of trusting a framework.
Other controls earned earlier and re-confirmed tonight: the BOS token (id 1) NaN-poisons layer 0
and must never be added; the model is a THINKING model — raw completion is out-of-distribution
(it predicted "Berlin" in the top-5 for "The capital of France is" — capitals-space, correct
neighborhood; clean Python predicts the identical top-5, proving this is model behavior, not an
engine bug); the chat template + <longcat_think> is the in-distribution format.
7. Generation: the dragon speaks
Tokenizer stays out of C: Python dumps vocab.bin (131,072 length-prefixed decoded strings) once;
C prints text directly. Prompt: the official chat template for "What is 2+2?" →
<longcat_user>What is 2+2? /think_on <longcat_assistant><longcat_think>.
First generated words from the from-scratch engine:
"The user is asking a simple arithmetic question"
Fluent, on-topic chain-of-thought. No NaN, no gibberish. This sentence has remained byte-identical
across every subsequent engine optimization — it is the correctness canary.
8. The speed war: three rounds, each measured, each a lesson
Round 1 — f32 LRU weight cache + MLA KV cache (dragon_fast.c): 88 s/token. FAILED to speed up.
The KV cache itself is correct (reproduces recompute output exactly — "The user…" token-for-token)
and kills the O(n²) attention recompute. But the LRU cached weights dequanted to f32, which
QUADRUPLES fp8 and DOUBLES bf16: the dense MLPs alone become 51 GB of f32 — cannot fit 31 GB —
41% hit rate — thrash. Measured diagnosis: the bottleneck was never attention. It is WEIGHT
BANDWIDTH — each token moves ~50 GB of weights.
Round 2 — native-precision compute (dragon_native.c, scalar): 27 s/token (3.3×).
Delete every f32 weight buffer. The matvec reads the mmap'd bf16/fp8 bytes DIRECTLY, dequanting
inline per element. Bytes moved drop 2–4×; the dense MLPs in native bf16 are 25 GB, which FITS —
so the OS page cache holds the hot working set and "load once" happens for free, in hardware.
Round 3 — the matmul, written properly (AVX2 + LUT): ~9 s/token (10× total), prefill 2.8 s/token.
fp8 via 256-entry lookup table. e4m3 has exactly 256 possible values. The scalar path was
paying a ldexpf LIBRARY CALL PER WEIGHT BYTE of a 560B model. Precompute all 256 floats once;
dequant becomes a table read (AVX2 vgatherdps, 8 lanes).
bf16 via AVX2+FMA. bf16→f32 is literally a 16-bit left shift — free inside SIMD:
load 8×u16 → widen → <<16 → that IS the f32 → fmadd. Two accumulators to hide FMA latency.
8 weights per instruction. The 131,072×6144 lm_head uses the same kernel.
128-column block structure of the fp8 scales folds the scale multiply out of the inner loop.
Round 4 — the GPU lights up (dragon_hybrid = dragon_native.c + dragon_gpu.cu): ~6 s/token (50× total).
Profiling showed the 9 s spread across attention 2.6 + dense 2.8 + MoE 3.7 + head 0.3 — and that the
box is fundamentally DISK-bound: the fixed per-token working set (~48 GB) exceeds 31 GB RAM, so every
token re-reads tens of GB from NVMe. The correct GPU move on an 8 GB card is therefore NOT "put the
model on the GPU" (impossible) and NOT "stream weights over PCIe" (16 GB/s — the trap): it is
VRAM as a permanent third cache tier. We park the attention weights — 42 of 56 sub-blocks,
7.5 GB bf16 — in VRAM at startup and never move them again. A warp-per-row bf16 matvec kernel
(one warp reduces one output row; __shfl_down_sync for the horizontal sum) computes attention
on-card; sub-blocks that don't fit fall back to the CPU path transparently. The win is DOUBLE:
attention drops 2.6 s → 0.5 s (5×), AND 7.5 GB leaves the RAM working set, so the page cache holds
more of the dense/expert tiers and disk reads shrink across the board.
engine
s/token
change
recompute (dragon_gen)
~300+
correct but O(n²), no caches
f32 LRU cache (dragon_fast)
88
KV cache ✓; f32 weight cache thrashes
native scalar (dragon_native)
27
zero f32 buffers; page cache = load-once
native AVX2+LUT
~9
the matmul
+ GPU attention tier (dragon_hybrid)
~6, falling
VRAM = 3rd cache tier; attn 5×
Output identical at every step — the generated sentence has never changed through a 50× speedup.
Correctness never traded for speed.
9. What this proves (the thesis, in working code)
Real answers = the war won. The existential question — can a 560B run correctly on owned
consumer hardware with owned code — is closed. Everything remaining is tuning, not truth.
The ScMoE shortcut is an owned moat. The architecture computes the MoE on block-0's output
and applies it at block-1's end — a designed overlap window. Generic runtimes don't exploit it;
an owned engine can compute MoE ∥ attention and hide expert-streaming latency inside attention
math. This optimization is only accessible to someone who OWNS the engine.
C, not Python — the game-engine discipline. No game runs Python in the frame loop; inference
IS a frame loop (tight, memory-bound, latency-critical). Python found the bugs (prototype);
C runs the frame (engine). The Python stack's own flakiness (silent meta-weights, wedged
processes, OOM kills) was demonstrated during this build and is documented above.
Bandwidth is the drivetrain. Every real speedup came from moving fewer bytes or moving them
smarter (native precision, page cache, SIMD) — never from more FLOPs. The datacenter's answer
to this problem is to buy more bandwidth. The drivetrain answer is to stop wasting it.
10. Remaining work (all de-risked, all speed/usability)
fp8-quantize the dense tier: 25 GB bf16 → 12.7 GB with our own (bit-exact-validated) fp8
path — brings the fixed working set under RAM so disk only ever reads experts. Largest remaining lever.
ScMoE overlap: thread the MoE against sub-block-1's attention (the moat, §9.2).
Compressed-latent KV cache: store the 576-dim latent instead of expanded K/V for long
context (matters beyond our current 64-token horizon).
src/dragon_native.c — CPU-only engine (no CUDA required)
src/dragon_forward.c — the proven full-forward validator
src/dragon_read.c, src/phase2*.c — the six validation bricks (reader, RMSNorm, fp8, MLA, MoE, full layer)
tools/build_manifest.py + Makefile — make dragon / make dragon-gpu
launch_card.png (the launch card) · dragon_speaks_card.png (the historic first-words shot) · manifest.bin (the 43,756-tensor index)
python3 tools/build_manifest.py /path/to/LongCat-Flash-Thinking-FP8 .
make dragon # CPU-only
make dragon-gpu # with the RTX 4070 attention tier
./dragon 16
— kapitoshkina.ai · bespoke ontology, 2026-07-05. This is the beginning.
