This is a randomly initialized, tiny debugging checkpoint with the
DeepSeek-V4 architecture. It is not a trained language model and must not be
used to evaluate model quality.
It was converted from
yujiepan/deepseek-v4-tiny-random
at revision 082d9c80d8e5b3f1b81e4a18e0a22daad84537af to exercise NVFP4 fused-MoE
kernels on NVIDIA Blackwell GPUs.
Quantization layout
All routed expert w1, w2, and w3 weights in the 7 transformer layers
were converted from MXFP4 group-32 to NVFP4 group-16.
The routed-expert tensors use the ModelOpt-compatible NVFP4 schema: packed
uint8 weights, FP8 E4M3 block scales, and scalar global/input scales.
Dense FP8 tensors, shared experts, embeddings, the LM head, and MTP tensors
are unchanged from the source mock checkpoint.
The model config uses activation="silu" and swiglu_limit=10.0, requiring
the clamped SwiGLU semantics:
This is therefore a mixed checkpoint intended specifically for routed-expert
NVFP4 testing, not a claim that every linear layer is NVFP4.
Validation
The checkpoint was tested on an NVIDIA GB10 (SM121) with:
vLLM nightly v0.26.1rc1.dev306+gcb8104839
FlashInfer 0.6.18 plus the B12X SiLU clamp fix
forced FLASHINFER_B12X NVFP4 MoE backend
activation="silu", swiglu_limit=10.0
All seven B12X MoE wrappers were constructed with the limit and an end-to-end
prefill plus two decode steps completed. The tested vLLM nightly predates the
B12X clamp capability declaration and parameter forwarding, so it required an
in-process adapter bridge. This is a runtime integration limitation, not part
of the checkpoint.
Some non-MoE DeepSeek-V4 tiny-shape paths also required reference fallbacks on
GB10. The checkpoint should be treated as an expert-kernel/debugging fixture,
not as a drop-in production serving model.
