Views
No views yet
trohrbaugh/Qwen3.5-122B-A10B-heretic (KL≈0.09)model_free RTN; shared-expert → FP8 E4M3 (block-128) from Qwen/Qwen3.5-122B-A10B-FP8OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4)max-num-seqs 1)| Config | tok/s |
|---|---|
MTP num_spec=2 | 44.8 |
MTP num_spec=3 | 44.5 |
| no MTP | 36.6 |
vllm-qwen35-v2). Also loads on stock vLLM as plain INT4 (slower, no FP8 shared-expert).vllm serve GaelicThunder/Qwen3.5-122B-A10B-heretic-hybrid-int4fp8 \
--max-model-len 262144 --gpu-memory-utilization 0.75 --trust-remote-code \
--load-format fastsafetensors \
--enable-chunked-prefill --max-num-batched-tokens 8192 --max-num-seqs 1 \
--reasoning-parser qwen3 --attention-backend FLASHINFER \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--enable-auto-tool-choice --tool-call-parser qwen3_xmlconfig.json sets quantization_config.block_name_to_quantize = "model.language_model.layers".
This is required: it scopes quantization to the main transformer blocks so the MTP module
(mtp.*) and vision tower stay unquantized. Without it, vLLM's qwen3_5_mtp loader raises
KeyError: 'layers.0.mlp.experts.w2_weight'.Qwen/Qwen3.5-122B-A10B — verify). Abliteration by trohrbaugh.
This model is uncensored and will comply with requests a safety-tuned model would refuse.
You are responsible for lawful, ethical use.