The model has trained-in always-thinks behavior — it emits <think>...</think> blocks regardless of any enable_thinking: false chat template flag. Do not try to suppress this with logit_bias on the special tokens; it corrupts generation (returns 1-token role-marker garbage like Action, assistant, Human and stops on prompts that engage the thinking pathway). An earlier version of this README recommended that workaround — it was wrong.
The correct architecture is to let the model emit thinking and strip the markup post-emit on a single content channel:
Drop --reasoning-parser qwen3 from your vLLM serve command. With it on, the parser greedily classifies all output before </think> as reasoning_content. If the model truncates without closing </think> (common in long agent loops), the entire answer ends up in reasoning_content and content is empty. See vllm-project/vllm#40816.
Strip <think>...</think> markup downstream in your gateway / client. Whether the model emits empty <think></think> (common on simple prompts) or a full thinking block, treat it as a uniform inline-markup case. Empty / whitespace-only blocks should be dropped silently. Closed blocks can be salvaged to a separate reasoning_content channel for observability if you want to surface thinking in your UI.
A reference implementation of this strip pipeline (LiteLLM custom callback) is at protoLabsAI/homelab-iac stacks/ai/config/litellm/callbacks/thinking_normalizer.py. Validated end-to-end with this model under the gateway-strip pipeline: clean output, no role-marker garbage, thinking captured to observability metadata, tool calls land cleanly outside <think> blocks.
--language-model-only — required for the Mamba/SSM hybrid architecture
--chat-template — use a Qwen3.5 nothink template (drops the <think>\n priming on assistant generation, even though the model will emit its own <think>...</think> anyway)
Notably absent:--reasoning-parser qwen3. See "How to handle thinking tokens" above.
Verified behavior
Under the gateway-strip pipeline (no logit_bias, no --reasoning-parser), tested 2026-05-01 on RTX PRO 6000 Blackwell with vLLM 0.20.0:
Probe
Result
Simple chat ("capital of France?")
13 tokens, <think></think>\n\nThe capital of France is **Paris**. — empty thinking block, gateway strips it, clean answer reaches client
Long-form prose (200-word internal monologue)
343 tokens, ends cleanly, no hedging, voice consistent with prompt
JSON schema compliance
Valid JSON with all required keys + correct types on first try
Multi-turn instruction-following
Coherent critique with continuity across turns
Tool-call (function calling)
~130 tokens of thinking before tool call, tool call lands outside <think> block, args clean
Tool-call thinking is a real latency cost (~130 tokens / ~500ms per turn through this model) but is a model-level behavior, not a quantization artifact — the official Qwen/Qwen3.6-35B-A3B-FP8 shows the same pattern.
Refusal calibration
The base model (llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) was trained to remove "rejections, objections, pushbacks, lecturing, censorship, softening and deflections" — its model card reports an 88% reduction in refusals vs the original Qwen/Qwen3.6-35B-A3B on a custom 100-prompt eval, with a KL divergence of 0.0015 from the original. This FP8 quant preserves that uncensored calibration on standardized refusal benchmarks.
XSTEST head-to-head (n=450, full dataset)
Metric
Qwen/Qwen3.6-35B-A3B-FP8 (official)
This model (heretic FP8)
Δ
Compliance on safe prompts
67.2%
88.8%
+21.6 pp
Over-refusal rate
31.2%
10.4%
−20.8 pp
Refusal on harmful prompts
99.5%
63.0%
−36.5 pp
Compliance on harmful prompts
0.5%
36.0%
+35.5 pp
The shift is bidirectional and consistent with the uncensored direction the upstream tune was designed for: less reflexive refusal on safe-but-sensitive prompts (over-refusal drops by ~21 pp) and less guardrail on harmful prompts (refusal rate drops by ~37 pp). FP8 quantization preserves both effects.
Methodology
Dataset: xstest — 250 safe prompts paired with 200 contrast unsafe prompts, designed to detect over-refusal vs proper refusal calibration.
Pipeline: gateway-strip (no --reasoning-parser, no logit_bias). Inline <think>...</think> markup stripped post-emit by a LiteLLM custom callback before responses reach the judge. Reference impl: thinking_normalizer.py.
Judge: Qwen/Qwen3.6-27B-FP8 (thinking enabled), classifying each (prompt, response) as comply / refuse / partial / unknown.
Sample sizes: heretic FP8 — n=450 (full dataset, 2026-05-02). Official Qwen3.6-35B-A3B-FP8 baseline — n=450 (full dataset, 2026-04-29, same harness).
Hardware: RTX PRO 6000 Blackwell, vLLM 0.20.0.
Note: this is a different methodology from the base model card's custom 100-prompt eval. The two numbers (88% reduction upstream vs the per-bucket shifts here) cannot be directly compared, but both point in the same direction. The per-bucket xstest numbers are the more transparent comparison since the official Qwen3.6-35B-A3B-FP8 was run against the same dataset, judge, and harness.
When this matters
Creative writing, red-team eval, scenario planning: the over-refusal drop (31.2% → 10.4%) means the model is less likely to refuse safe-but-sensitive prompts (medical, legal, historical, fictional violence). This is the value prop.
General-purpose assistant deployment: the harmful-prompt compliance shift (0.5% → 36.0%) is a real safety surface change. Self-harm category in particular is high-compliance (89% in spot checks). If exposing this model to general traffic, plan for category-level guardrails (gateway-side or system-prompt-level) rather than relying on the model's own refusal calibration.
Known Issues
Always thinks: trained-in behavior — see "How to handle thinking tokens" for the correct mitigation.
Broken vision: image inputs produce degenerate output (repeated !!!! characters), inherited from the base model. Use --language-model-only and do not pass image content.
Quantization Details
Method: Block-wise FP8 E4M3 with [128, 128] per-block scaling
MoE handling: Packed 3D expert tensors unpacked to per-expert 2D weights; fused gate_up_proj split into separate gate_proj + up_proj
Preserved in BF16: Embeddings, LM head, LayerNorm, MoE router gates, Mamba/SSM parameters (conv1d, A_log, D, dt_bias, in_proj), shared expert gates — 402 modules total via modules_to_not_convert
Quantization Script
The quantization script (quantize_native_fp8_lowmem.py) is included in this repo. It processes safetensors shards one tensor at a time, peaking at ~4GB RAM regardless of model size.
