⚠️ Disclaimer: Entirely Vibe Coded! This is my first model experiment. Don't expect highest quality — built for the NVIDIA DGX Spark GB10, your mileage may vary.
NVFP4 (NVIDIA FP4) is a Blackwell-native 4-bit floating-point format supported by the GB10's Tensor Cores. It gives roughly 2× memory reduction vs bf16 with near-zero quality loss, and enables the full 40B model to fit comfortably in the GB10's 128 GiB unified memory.
DFlash note:DFlash (arXiv:2602.06036) is a block-diffusion speculative decoding method where a small 5-layer draft model predicts multiple tokens in parallel. The z-lab 27B drafter was trained against a 27B target's hidden states. Using it with this 40B model gives ~13.6% token acceptance (vs ~32% for a matched 27B→27B pair) — it still works and provides a real speedup, but a drafter trained specifically for the 40B target would be significantly better. If you have the resources to train one, please reach out!
Benchmark Results
All benchmarks run on a NVIDIA DGX Spark GB10, client on a separate x86 machine. B=N = N concurrent requests. HumanEval=20 problems, MMLU-STEM=50 questions, Tool Use=20 tasks. The 27B AEON rows use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-NVFP4 and are included as reference baselines.
⚠️ These are minimal smoke tests to get a directional sense of performance — sample sizes are small and results are not statistically representative. Treat the numbers as rough indicators, not rigorous benchmarks.
Model
KV Cache
B=1¹ (tok/s)
B=4¹ (tok/s)
B=8¹ (tok/s)
DFlash accept. rate
HumanEval
MMLU
Tool Use
27B AEON solo (NVFP4)
bf16
12.7
51.0
81.1
—
65%
98%
80%
27B AEON solo
fp8_e4m3
12.8
51.2
81.4
—
—
—
—
27B AEON solo
turboquant k8v4
12.7
50.8
80.6
—
—
—
—
27B AEON + DFlash (matched)
bf16
50.3
140.3
178.4
32.1%
65%
98%
80%
40B Deckard solo (NVFP4)
bf16
6.5
24.4
40.2
—
85%
98%
90%
40B Deckard solo
fp8_e4m3
6.6
24.2
39.6
—
—
—
—
40B Deckard solo
turboquant k8v4
6.5
24.0
33.6
—
—
—
—
40B Deckard + z-lab 27B DFlash†
bf16
16.0
38.8
50.2
13.6%
90%
100%
95%
¹ Output tokens/sec at N concurrent requests (single-user baseline → light multi-user load). Higher concurrency increases GPU utilization and amortizes fixed overhead, so throughput scales non-linearly.
†Mismatched drafter — z-lab's drafter was trained against a 27B target, not the 40B. A matched 40B drafter would push acceptance to ~32% and throughput to 4×+ at B=1.
Observations:
The 40B scores measurably higher than the 27B on all quality benchmarks — 85–90% HumanEval vs 65%, 90–95% Tool Use vs 80%. The Claude 4.6 Opus fine-tuning and parameter expansion are visible in the numbers.
DFlash is lossless — 40B+DFlash quality matches or exceeds the 40B solo across all three benchmarks.
DFlash was only tested with bf16 KV cache. The flash_attn fp8 kernels used by DFlash target SM89/SM90 and are incompatible with the GB10 (SM12.1a), ruling out fp8_e4m3 KV in the current stack. Combining DFlash with turboquant k8v4 KV quantization would be an interesting further optimization but was not tested here.
fp8_e4m3 KV cache has negligible throughput impact (±0.2%) with 2× KV memory saving.
turboquant k8v4 reduces throughput at B=8 (−16% on the 40B) with no quality benefit.
The matched 27B→27B DFlash (32.1% acceptance, 178.4 tok/s at B=8) shows how much headroom remains once a proper 40B-trained drafter exists.
Test scripts used are uploaded to this repo (eval/).
Both models use a hybrid architecture (full_attention_interval=4): only every 4th layer is full attention; the rest are linear (Mamba-style) attention with O(1) state. This means only 25% of layers need a traditional KV cache — dramatically less than a pure transformer.
NVFP4 safetensors store each FP4 value in an FP8 container (~1 byte/param on disk and in RAM); actual FP4 compression happens at compute time in Blackwell Tensor Cores
Measured weight sizes from HF: 24.6 GiB (27B) / 35.9 GiB (40B, this repo)
DFlash drafter (z-lab/Qwen3.6-27B-DFlash): 3.2 GiB of weights (bf16; 5 layers + fc + norms). It shares the target's token embedding and LM head, so those add no extra memory. It also keeps a small KV cache of its own during speculation (~5 GiB at full 256k context, included in the DFlash rows below).
DFlash drafter KV uses its own geometry (5 layers, 8 kv_heads, head_dim 128) → ~5 GiB at 256k
Max concurrent assumes model weights are shared across requests (vLLM default), 6 GiB reserved for Xorg/system on GB10
A pure transformer with the same layer count would need 4× more KV cache — the hybrid architecture is a key reason this 40B model is practical on the GB10.
Files in This Repo
File
Description
config.json
Model architecture config (Qwen3_5, compressed-tensors NVFP4)
recipe.yaml
llmcompressor quantization recipe used to create this model
No matched DFlash drafter for 40B. The z-lab drafter targets 27B hidden states. Training a proper 40B drafter (cfg6 in my notes) requires reproducing the DFlash training recipe (arXiv:2602.06036 §4.2) — feasible but compute-intensive. Happy to collaborate if you have the hardware.
Blackwell-only. NVFP4 requires SM90+ (GB10/H100/H200/B200). Will not run on Ampere or older.
CUDA memory note. On the GB10's unified memory, each container start/stop leaks ~6 GiB at the driver level. A full reboot between model swaps is recommended.
