You can deploy this model and start earning money today!

IKilledBardia/
Qwen3.6-27B-FP8-unsloth

Name: Qwen3.6-27B-FP8-unsloth
Availability: InStock

Views

No views yet

Qwen3.6-27B-FP8 (unsloth) — block-wise FP8

FP8 quantization of Qwen/Qwen3.6-27B (a hybrid linear-attention + full-attention model), ready for vLLM.

Format

Weights: FP8 (float8_e4m3fn), block-wise [128, 128] scales (weight_scale_inv).
Activations: FP8 dynamic (per-token) — no calibration needed.
quant_method: fp8, weight_block_size: [128,128], activation_scheme: dynamic.
Vision tower, lm_head, embeddings, norms, and the SSM parameters (conv1d, in_proj_a/b, A_log, dt_bias) are kept in higher precision.

MMLU (lm-eval, `--tasks mmlu`, 5-shot, full test set, vLLM)

Model	MMLU acc
This model (block-wise FP8)	0.8445 ± 0.0030
Qwen/Qwen3.6-27B-FP8 (reference)	0.8428 ± 0.0030

Per-task and raw results are in eval/.

Usage (vLLM)

python

from vllm import LLM
llm = LLM(model="IKilledBardia/Qwen3.6-27B-FP8-unsloth", trust_remote_code=True)

Requires a vLLM build that supports Qwen3_5ForConditionalGeneration and is compiled against your CUDA runtime (e.g. vLLM 0.19.1 for CUDA 12.8).

History

An earlier revision of this repo used per-tensor modelopt FP8. Although its stored weights were numerically faithful to the base model, vLLM's qwen3_5 execution path mishandles the per-tensor FP8 format for this hybrid architecture, collapsing MMLU to random (0.2325). Re-exporting in the block-wise [128,128] FP8 format above restores full quality.

IKilledBardia/Qwen3.6-27B-FP8-unsloth