You can deploy this model and start earning money today!

mconcat/
Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic

Name: Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic
Availability: InStock

Views

No views yet

Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic

FP8_DYNAMIC quantized version of Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2.

This checkpoint keeps the same hybrid Qwen3.5 DeltaNet + softmax architecture and Qwen3.5 MTP head as the BF16 source, but quantizes most linear layers to FP8 W8A8 while leaving the most sensitive projections and sidecar components in BF16.

The published folder includes:

model.safetensors
model.safetensors.index.json
model.mtp.safetensors
processor_config.json
preprocessor_config.json
video_preprocessor_config.json
recipe.yaml

Verified Inference

Local export and sanity-check evaluation were verified on 2026-03-31 on a single NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) with:

transformers==5.3.0
llm-compressor==0.10.1.dev40+g5ae2e149
vllm==0.17.1

What was verified in that run:

the FP8 export completed successfully
model.mtp.safetensors was restored into the output folder
the checkpoint loads in transformers with device_map="auto"
a quick perplexity sanity check against the BF16 source completed successfully

vLLM is still the intended serving path for this family of models, but full local serve validation for this exact FP8 v2 export is still pending.

Quantization Strategy

Uniform FP8_DYNAMIC quantization using llm-compressor:

Precision	Layers
FP8 W8A8	most `Linear` layers
BF16	`lm_head`, `embed_tokens`, `self_attn.o_proj`, DeltaNet `linear_attn.out_proj`, DeltaNet `in_proj_a`/`in_proj_b`, visual encoder, MTP sidecar

FP8 details:

weights: FP8, per-channel, static scales
input activations: FP8, per-token, dynamic scales
output activations: not explicitly quantized

Architecture match with the BF16 source:

model_type=qwen3_5
64 text layers
full_attention_interval=4
mtp_num_hidden_layers=1
max_position_embeddings=262144

Local Benchmark Slice

Local quick sanity comparison against the BF16 v2 source model, run on 2026-03-31 with a small FineWeb-Edu perplexity slice.

This is a reduced verification slice, not a full benchmark run.

Benchmark	BF16 v2	FP8 v2
FineWeb-Edu perplexity (20 samples, 13,531 tokens, max_len=1024)	7.0758	7.1051

Absolute perplexity delta:

+0.0293
about +0.41% relative to BF16

Usage

vLLM

pip install -U vllm==0.17.1 transformers==5.3.0

Expected serving command:

bash

vllm serve mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 1 \
  --skip-mm-profiling \
  --reasoning-parser qwen3

With MTP enabled:

bash

vllm serve mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --kv-cache-dtype fp8_e4m3 \
  --max-num-seqs 1 \
  --skip-mm-profiling \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Transformers

python

from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
import torch

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    "mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    "mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-FP8-Dynamic",
    trust_remote_code=True,
)

Compatibility

Framework	Supported	Notes
vLLM >= 0.17.0	Expected	Intended serving path; this exact v2 FP8 export still needs full local serve re-validation
transformers >= 5.3.0	Yes	Direct loading works with `device_map="auto"`
SGLang	Unknown	Not verified for this export

Notes

This export keeps self_attn.o_proj and DeltaNet linear_attn.out_proj in BF16 rather than FP8.
The output folder includes the Qwen3.5 MTP sidecar and processor metadata needed for serving compatibility.
The perplexity numbers above are intended as a quick sanity check, not as an exhaustive benchmark submission.