Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8
🇬🇧 English
FP8 (block-128 e4m3) + native MTP build of
Qwen3.6-27B-DSV4Pro-Thinking-Distill — for
SGLang / vLLM serving (especially concurrent serving), where FP8 tensor cores + batching shine.
For the
distillation method, attribution, eval protocol and limitations, see the
parent BF16 model card. This card only covers the FP8 build.
What this is
- FP8 block-128, e4m3, dynamic activation quantization of the distilled trunk — same scheme as the official
Qwen/Qwen3.6-27B-FP8 (weight_scale_inv per 128×128 block). Norms, embeddings, lm_head, the small Gated-DeltaNet projections, and the vision tower are kept BF16 (matching the official layout).
- Native MTP (nextn) head bundled (
mtp.safetensors) for EAGLE/NEXTN speculative decoding.
- Near-lossless by construction — block-128 FP8 is the same precision the Qwen team ships as "performance nearly identical to the original".
Speed — distill FP8 measured on DGX Spark (GB10, Blackwell sm_121), SGLang dev-cu13 + flashinfer + EAGLE/NEXTN
| Mode | Throughput | MTP accept rate | accept length |
|---|
| single-stream, no MTP (base) | ~7.7 tok/s | — | — |
| single-stream, MTP on | ~15 tok/s (→ ~19 on code) | 0.65–0.89 | 2.6–3.55 |
| concurrent ×16 — official-FP8 ref | ≈136–143 tok/s aggregate | — | — |
MTP gives ~2× single-stream (base 7.7 → 15 tok/s, up to ~19 / accept 0.89 on predictable code). All measured on this distilled FP8 (SGLang dev-cu13, EAGLE topk=1 / 3 draft / mamba extra_buffer); generation correct, no FP8 corruption. The concurrent figure is from the structurally-identical official Qwen/Qwen3.6-27B-FP8 (distill concurrent not separately benchmarked).
Where FP8+MTP wins: single-stream on a bandwidth-bound box is not its arena (a Q4_K_M-imatrix GGUF is faster there ~25 tok/s) — FP8+MTP's strength is concurrent serving: FP8 tensor cores + batching + near-lossless quality. On Blackwell RTX-50 (faster FP8) it's faster still.
Quality — formal same-harness FP8 eval
Both this distilled FP8 and the official base FP8 (Qwen/Qwen3.6-27B-FP8) were run through the identical streaming harness on DGX Spark (SGLang dev-cu13, thinking-on, temperature 0.6 / top_p 0.95, context 36864): GPQA-Diamond full-198 and MMLU-500 5-shot.
| Dimension | Distill FP8 | Base FP8 | Δ |
|---|
| GPQA-Diamond-198 | 82.32% (163/198) | 66.67% (132/198) | +15.65 |
| MMLU-500 (5-shot) | 90.00% (450/500) | 88.60% (443/500) | +1.40 |
GPQA finish=length (runaway) | 0 | 12 | — |
The distilled FP8 is +15.65 pts on GPQA over the un-distilled base — and the headline is convergence: 0 length-truncations vs the base's 12. On hard GPQA items the un-distilled base lets its chain-of-thought run away until it slams into the token ceiling (those 12 unfinished answers are scored wrong); the distill was trained to rein exactly that in, while staying ahead on MMLU too. FP8 itself is near-lossless — numbers track the parent BF16 distill (canonical GPQA 80.81 / MMLU 91.8). Prior canonical reference (not re-run under FP8): coding-100 86 vs 83, Agentic SOLO-20 16 vs 13.
Serving (SGLang, validated launch)
1SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \
2 --model-path Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
3 --speculative-algorithm EAGLE --speculative-num-steps 3 \
4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
5 --mamba-scheduler-strategy extra_buffer \
6 --attention-backend flashinfer --trust-remote-code
- Requires SGLang ≥ 0.5.10 (Qwen3.6). The
extra_buffer mamba scheduler + SPEC_V2 are required for speculative decoding on the Gated-DeltaNet (mamba-style) layers. On Blackwell use --attention-backend flashinfer.
- thinking-on,
temperature=0.6, top_p=0.95 (never greedy).
Files
model-*-of-trunk.safetensors — FP8 (e4m3, block-128) distilled trunk + BF16 norms/embed/lm_headmtp.safetensors — native nextn MTP head (FP8)model-vision.safetensors — vision tower (BF16; unused for text)config.json carries the quantization_config (fp8 e4m3)
🇨🇳 中文版
Qwen3.6-27B-DSV4Pro-Thinking-Distill 的
FP8(block-128 e4m3)+ 原生 MTP 版 —— 面向
SGLang / vLLM 服务化(尤其并发 serving),FP8 张量核 + 批量在这里发挥。
蒸馏方法、出处归因、评测口径、局限见
父模型(BF16)卡片。本卡只讲 FP8 版。
这是什么
- FP8 block-128、e4m3、动态激活量化蒸馏主干 —— 与官方
Qwen/Qwen3.6-27B-FP8 同方案(每 128×128 块一个 weight_scale_inv)。norm、embedding、lm_head、Gated-DeltaNet 小投影、vision 塔保持 BF16(对齐官方布局)。
- 打包原生 MTP(nextn)头(
mtp.safetensors),供 EAGLE/NEXTN 投机解码。
- 构造上近无损 —— block-128 FP8 正是 Qwen 官方"性能与原模型几乎一致"的同精度。
速度 —— 蒸馏 FP8 实测 DGX Spark(GB10,Blackwell sm_121),SGLang dev-cu13 + flashinfer + EAGLE/NEXTN
| 模式 | 吞吐 | MTP 接受率 | 接受长度 |
|---|
| 单流,关 MTP(base) | ~7.7 tok/s | — | — |
| 单流,开 MTP | ~15 tok/s(代码段 → ~19) | 0.65–0.89 | 2.6–3.55 |
| 并发 ×16 —— 官方 FP8 参考 | ≈136–143 tok/s 聚合 | — | — |
MTP 单流约 2× 加速(base 7.7 → 15 tok/s,代码这类可预测内容升到 ~19 / 接受率 0.89)。均为本蒸馏 FP8 实测(SGLang dev-cu13,EAGLE topk=1 / 3 draft / mamba extra_buffer):生成正确、无 FP8 损坏。并发数来自结构一致的官方 Qwen/Qwen3.6-27B-FP8(蒸馏版并发未单独实测)。
FP8+MTP 的主场:在带宽受限的机器上单流不是它的强项(那里 Q4_K_M-imatrix GGUF 更快 ~25 tok/s)—— 它强在并发 serving:FP8 核 + 批量 + 近无损质量。Blackwell RTX-50(FP8 更快)上更快。
质量 —— 同 harness 正式 FP8 评测
本蒸馏 FP8 与官方 base FP8(Qwen/Qwen3.6-27B-FP8)跑完全相同的流式 harness(DGX Spark,SGLang dev-cu13,thinking-on,temperature 0.6 / top_p 0.95,context 36864):GPQA-Diamond 全 198 + MMLU-500 5-shot。
| 维度 | 蒸馏 FP8 | Base FP8 | Δ |
|---|
| GPQA-Diamond-198 | 82.32% (163/198) | 66.67% (132/198) | +15.65 |
| MMLU-500 (5-shot) | 90.00% (450/500) | 88.60% (443/500) | +1.40 |
GPQA finish=length(跑飞) | 0 | 12 | — |
蒸馏 FP8 在 GPQA 上 +15.65 分,而最亮的是收口:0 次长度截断 vs base 的 12 次。难题上未蒸馏的 base 思维链收不住、一路撞到 token 上限(那 12 个没答完的判错);蒸馏正是为修这个而训,同时 MMLU 也保持领先。FP8 本身近无损——数据贴合父 BF16 蒸馏版(canonical GPQA 80.81 / MMLU 91.8)。早前 canonical 参考(未在 FP8 下重跑):coding-100 86 vs 83,Agentic SOLO-20 16 vs 13。
服务化(SGLang,已验证启动)
1SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \
2 --model-path Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
3 --speculative-algorithm EAGLE --speculative-num-steps 3 \
4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
5 --mamba-scheduler-strategy extra_buffer \
6 --attention-backend flashinfer --trust-remote-code
- 需 SGLang ≥ 0.5.10(Qwen3.6)。
extra_buffer mamba 调度 + SPEC_V2 是 Gated-DeltaNet(mamba 式)层做投机解码的必需项;Blackwell 用 --attention-backend flashinfer。
- thinking-on,
temperature=0.6, top_p=0.95(切勿 greedy)。
文件
model-*-of-trunk.safetensors —— FP8(e4m3, block-128)蒸馏主干 + BF16 norm/embed/lm_headmtp.safetensors —— 原生 nextn MTP 头(FP8)model-vision.safetensors —— vision 塔(BF16,文本不用)config.json 带 quantization_config(fp8 e4m3)
Claude Code（实验性 / experimental）
Claude Code 不是本地 GGUF/HF 加载器——它对接兼容 Anthropic /v1/messages 的后端、且要求可靠的工具调用，不能直接喂仓名/路径。本 FP8 仓是 vLLM 路线的直接载体：
1vllm serve nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-FP8 \
2 --served-model-name qwen36-27b-fp8 \
3 --reasoning-parser qwen3 \
4 --enable-auto-tool-choice \
5 --tool-call-parser qwen3_xml
Claude Code 里模型名填
qwen36-27b-fp8（即
--served-model-name，
不要填带 / 的仓名），
ANTHROPIC_BASE_URL 指向你的 vLLM 端点。要 GGUF + LM Studio 路线见
GGUF 仓。参考
vLLM Claude Code。
Claude Code (experimental)
Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible
/v1/messages backend and needs reliable tool calling, so you cannot point it at a repo id directly. This
FP8 repo is the direct vLLM path — serve it with
--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 (command above), then set Claude Code's model name to the
--served-model-name (no slashes) and
ANTHROPIC_BASE_URL to your vLLM endpoint. For the GGUF + LM Studio route, see the
GGUF repo.
