Block-wise FP8 (e4m3, 128x128 blocks, dynamic activation scaling) quantization of barozp/Qwen3.8-27B-Opus-Distill-v2, built with the exact recipe Qwen/Qwen3.8-27B-FP8 uses for the base model -- read live from that repo's quantization_config at build time, not guessed. No calibration dataset is needed (dynamic activation scheme): only weights are quantized, block-wise, from their own value range. Roughly 56% of the source bf16 size (~31 GB vs ~55 GB).

Excluded from FP8 (kept in original precision, same policy as Qwen's own FP8 release): vision tower, MTP head, norm/gate layers, embed_tokens, lm_head, and the Gated-DeltaNet SSM-specific parameters (A_log, conv1d, dt_bias, internal norm). Everything else (attention QKVO on full-attention layers, FFN, Gated-DeltaNet's own in_proj/out_proj matrices) is quantized.

Changelog

2026-08-21 -- IMPORTANT config fix (revision 67ce584). If you pulled this repo before that date and sglang fails to start with ValueError: Weight output_partition_size = 48 is not divisible by weight quantization block_n = 128, delete your local copy / re-download so you get the fixed config.json. Root cause: the modules_to_not_convert list must use bare module paths (...linear_attn.in_proj_ba) -- sglang resolves fused GDN layers through their shard names (in_proj_b, in_proj_a), so tensor-name-style entries (....weight) never match and a merged projection got FP8'd despite being bf16 in the checkpoint. Weights were always correct and are unchanged.
2026-08-22: model card rewritten with native-FP8 serving verification and serving guide.

Validation status

What	Where	Result
Structural integrity	build time	Zero NaN/Inf; vision/MTP tensors byte-identical to the bf16 source; size ratio as expected
`transformers` load + generation	A100 (auto-dequantizes to bf16 below compute capability 8.9)	Coherent output; token counts matched the bf16 model on a reasoning-loop repro -- i.e. validated on dequantized-bf16 compute, not native FP8
Native FP8 execution under sglang	RTX PRO 6000 Blackwell Server Edition (sm_120), sglang nightly cu129 stack	Full construction + all 1599 tensors loaded; server came up healthy across five consecutive runs; a real chat completion returned generated text with default sampling parameters
MTP / speculative decoding (NEXTN)	same GPU	Draft head (`Qwen3_5ForCausalLMMTP`, fp8) loads and serves; the internal warmup request completes through the draft/verify pipeline
Parity check	same GPU	Qwen's own official FP8 release behaves identically through the identical pipeline (same stages, same failure points when the test GPU's environment interferes)

Known environment limitation (not specific to this checkpoint): flashinfer's JIT arch-check currently rejects consumer Blackwell (sm_120) and bites in up to three places: attention planning during CUDA-graph capture, top-k/top-p sampling at request time, and EAGLE verify sampling during speculative decoding. Workarounds are documented below. One consequence on sm_120 specifically: a fully non-greedy MTP-accelerated user request cannot be exercised there today (the EAGLE verify sampler hard-wires flashinfer via sgl_kernel); greedy warmup requests do pass. On Ada/Hopper (sm_89/sm_90) flashinfer works normally and none of these paths are affected. vLLM could not be independently verified in our test image for dependency-conflict reasons unrelated to this checkpoint; the format matches the standard HF FP8 layout used by Qwen's own releases.

Paired BF16-vs-FP8 benchmarks

Both checkpoints were evaluated under an identical protocol so the delta isolates true quantization cost from any serving-stack numerics:

Task	Metric	BF16 source	FP8 build	Δ (FP8−BF16)
MMLU	acc	0.8474	0.8462	−0.0012
HellaSwag	acc_norm	0.7520	0.7480	−0.0040
ARC-Challenge	acc_norm	0.6240	0.6180	−0.0060

Every delta sits inside one standard error (~±0.02): the FP8 build's quality cost on these tasks is statistically indistinguishable from zero. Sanity check: our BF16 rerun reproduced this card's published reference values almost exactly (0.8476 / 0.7500 / 0.6220), confirming both the card and the protocol.

Test environment (reproduction)

Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition (sm_120), 97 GB
Software: lm-evaluation-harness 0.4.12 · transformers 5.15.0 · torch 2.11.0+cu128
Backend: lm-eval HF backend (--model hf) — no serving layer, no batching nondeterminism
dtype bfloat16 for both checkpoints · chat template OFF · 0-shot (--num_fewshot 0)
--limit 500 (QUICK mode) · --batch_size 16 · --seed 1234

Command shape (run once per checkpoint):

bash

python3 -m lm_eval \
    --model hf \
    --model_args pretrained=<CHECKPOINT>,dtype=bfloat16,trust_remote_code=True \
    --tasks mmlu,hellaswag,arc_challenge \
    --num_fewshot 0 \
    --batch_size 16 \
    --limit 500 \
    --seed 1234 \
    --output_path <OUT_DIR>

Notes:

The FP8 checkpoint executes through native FP8 matmul in transformers 5.x, which needs pip install "kernels==0.16.0" — otherwise you'll hit ImportError: finegrained-fp8 kernel unavailable at the first FP8 linear layer.
Wikitext perplexity is excluded from this pairing: its rolling-loglikelihood evaluation triggers O(seq²) decay-mask allocation in this architecture's pure-torch fallback path and OOMs even on 97 GB GPUs.

Serving with sglang

Standard launch on Ada/Hopper (sm_89 / sm_90 -- e.g. RTX 4090, RTX 6000 Ada, H100):

bash

python3 -m sglang.launch_server \
    --model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \
    --trust-remote-code

Consumer Blackwell cards (sm_120: RTX PRO 6000, RTX 5090) currently need three workaround flags until flashinfer fixes its sm_120 detection:

bash

python3 -m sglang.launch_server \
    --model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \
    --trust-remote-code \
    --attention-backend triton \
    --disable-cuda-graph \
    --sampling-backend pytorch

Optional MTP / speculative decoding (NEXTN) -- add to either command above:

bash

    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

(If you are on sm_120, include the three workaround flags here too.)

Quick start (transformers)

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "barozp/Qwen3.8-27B-Opus-Distill-v2-FP8", dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill-v2-FP8")

On GPUs below compute capability 8.9, transformers will automatically dequantize to bf16 at load time (you'll see a warning) -- this still works correctly, it just won't give you the FP8 memory/speed benefit.

barozp/Qwen3.8-27B-Opus-Distill-v2 -- source model, full card.
barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF -- GGUF quantizations (BF16 down to IQ1_M) for llama.cpp.
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit and MLX-8bit -- Apple Silicon MLX conversions via mlx-vlm.
Qwen/Qwen3.8-27B-FP8 -- the base model's official FP8 release, whose recipe this follows.

barozp/Qwen3.8-27B-Opus-Distill-v2-FP8

Qwen3.8-27B-Opus-Distill-v2-FP8

Changelog

Validation status

Paired BF16-vs-FP8 benchmarks

Test environment (reproduction)

Serving with sglang

Quick start (transformers)

Related

barozp/
Qwen3.8-27B-Opus-Distill-v2-FP8