Near-lossless versus bf16. SWE-bench Lite resolves land within one instance of the
bf16 source at every size, the 27B lm-eval gap averages under 1.5 points, and
wikitext-2 perplexity for this build is 6.89. See Benchmarks.
About 25.6 GB on disk versus about 73.7 GB for the bf16 source (about 35%).
Built for vLLM on NVIDIA Blackwell, where both the 4-bit weight and 4-bit
activation paths are accelerated. On pre-Blackwell GPUs vLLM runs it weight-only.
Loading and generation verified in vLLM on an NVIDIA GB10 (Blackwell, sm_121).
Uncensored model. This is a quantization of an uncensored / abliterated
derivative. It follows instructions without content guardrails, including NSFW.
Behaviour and alignment are inherited entirely from the base model.
Benchmarks
benchmarks
Near-lossless versus the bf16 source:
SWE-bench Lite (agentic, mini-swe-agent, instances 0:20): resolves land within
one instance of bf16 at every size (NVFP4 15/13, bf16 16/14 of 20).
lm-eval (27B pair, the clean apples-to-apples): average accuracy gap under 1.5
points.
Full head-to-head tables and method in BENCHMARKS.md.
Fidelity
Near-lossless versus the bf16 source, 25.6 GB vs 73.7 GB bf16 (~35%), at wikitext-2 perplexity 6.89. See Benchmarks for the full head-to-head. GPTQ error compensation and an MSE observer keep the drop from bf16 minimal; the header lists the full characteristics and Quantization covers the recipe.
Quickstart
Offline (vLLM)
NVFP4 activation acceleration needs a Blackwell-class GPU. The if __name__ == "__main__" guard is required for offline LLM(...) because the vLLM v1 engine
spawns workers.
python
1from vllm import LLM, SamplingParams
23defmain():4 llm = LLM(5 model="maci0/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NVFP4",6 max_model_len=16384,7)8 msgs =[{"role":"user","content":"Write the opening paragraph of a noir short story."}]9 sp = SamplingParams(temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048)10 out = llm.chat(msgs, sp)11print(out[0].outputs[0].text)1213if __name__ =="__main__":14 main()
Server (OpenAI-compatible)
Recommended baseline for a single Blackwell GPU. The NVFP4 quantization is
auto-detected from config.json (compressed-tensors), so no quantization flag is
needed. --reasoning-parser qwen3 splits the <think> block into a separate
reasoning_content field.
These parser flags are not auto-detected; you must pass them explicitly. Drop the
last line if you do not need tool calling; --enable-auto-tool-choice requires
--tool-call-parser.
--media-io-kwargs '{"video":{"num_frames":-1}}' (and raise longest_edge in video_preprocessor_config.json)
Context notes:
The model supports up to 262144 tokens. Upstream guidance is to keep at least
128K to preserve thinking quality, so --max-model-len 131072 is the recommended
default. Go to 262144 if memory allows, or lower it if you hit OOM.
On unified-memory parts (e.g. GB10), --gpu-memory-utilization carves from RAM
shared with the rest of the system. Use about 0.90 when this is the only model,
and leave more headroom (about 0.80) when co-hosting other processes.
Python (OpenAI client)
python
1from openai import OpenAI
2client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")3r = client.chat.completions.create(4 model="qwen3.6-40b-nvfp4",5 messages=[{"role":"user","content":"Write the opening paragraph of a noir short story."}],6)7print(r.choices[0].message.content)
curl
bash
1curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
2 "model": "qwen3.6-40b-nvfp4",
3 "messages": [{"role": "user", "content": "Write the opening paragraph of a noir short story."}]
4}'
KV-cache quantization
The NVFP4 here quantizes weights and activations, not the KV cache (the checkpoint
ships kv_cache_scheme: null). KV-cache quantization is a separate runtime vLLM
option. Only the 24 full-attention layers hold a standard KV cache; the gated
delta-net linear-attention layers use recurrent state and are unaffected.
FP8 KV cache (recommended, safe): --kv-cache-dtype fp8 (or fp8_e4m3). About
2x KV savings with small quality cost. Used in the baseline command above.
TurboQuant (vLLM, experimental here): lower-bit KV quant via Hadamard
rotation plus per-coordinate Lloyd-Max scalar quantization. Values:
turboquant_k8v4 (FP8/4-bit, 2.6x, +1.17% PPL), turboquant_4bit_nc (3.8x),
turboquant_k3v4_nc (~3.5x), turboquant_3bit_nc (4.9x). turboquant_k8v4 is
the quality sweet spot. Caveat: TurboQuant uses a dedicated attention backend
whose interaction with this model's linear-attention layers was not verified.
Treat as experimental; prefer fp8 for a known-good KV quant.
Performance / backend notes (verified on vLLM)
FlashInfer is bundled and autotune is on by default. It is used automatically for
the full-attention and NVFP4 GEMM paths on Blackwell; there is nothing to enable.
Optional: VLLM_USE_FLASHINFER_SAMPLER=1 for faster sampling.
NVFP4 GEMM auto-selects cutlass FP4 on Blackwell. Do not set
VLLM_NVFP4_GEMM_BACKEND (deprecated in 0.23.0). Leave
VLLM_USE_NVFP4_CT_EMULATIONS=0 (the default; emulation is for pre-Blackwell).
Attention backend: leave on auto. This is a hybrid model, so vLLM assigns the
per-layer backends (GDNAttentionBackend / LinearAttentionBackend)
automatically. Forcing a single global attention backend breaks the
linear-attention layers.
No sparse-attention knob applies. The efficiency comes from the hybrid 3:1
linear:full attention layout, handled automatically.
About the base model
A 40B dense (not MoE) vision-language model expanded from Qwen3.6-27B, made
uncensored via Heretic, trained on the internal Deckard/PKD datasets (character,
depth, point of view) and on a Claude 4.6 Opus high-reasoning distillation set to
sharpen and stabilize reasoning.
96 decoder layers: hybrid gated delta-net linear attention (72) plus full
attention (24), dense MLP, plus a vision tower for image and video input.
256K context (max_position_embeddings 262144).
Thinking mode by default (variable-length reasoning), with an instruct toggle.
all language-model Linear layers, 744 modules (360 linear-attn projections + 288 MLP + 96 full-attn)
Kept in bf16
vision tower (model.visual.*), lm_head
Untouched
gated delta-net Conv1d and SSM params (A_log, dt_bias), not Linear, never targeted
GPTQ is a quantization-time cost only. The output is the same
nvfp4-pack-quantized format with identical inference speed; GPTQ just chooses
better 4-bit values than plain round-to-nearest.
Calibration
512 samples, domain-matched to the model's actual traffic, max_seq_len=2048,
text-only path through the VL model:
source
samples
domain
TeichAI/claude-4.5-opus-high-reasoning-250x
250
long reasoning (the base model's own training data)
HuggingFaceH4/ultrachat_200k
150
general chat
m-a-p/Code-Feedback
112
code
Quality
GPTQ with the domain-matched calibration measurably beats plain round-to-nearest. The
fused layers (q/k/v, gate/up) share one NVFP4 global scale, so vLLM does not warn or
fall back. Measured wikitext-2 perplexity for this build is 6.89 (see
Benchmarks); the gain is expected to be larger on the model's own
domains (reasoning, creative, code), which wikitext does not cover.
If the model loops on thin prompts, add a one-line system prompt (e.g. Be vivid and precise.) and/or set repetition_penalty 1.05 to 1.1.
To run instruct (non-thinking), set {%- set enable_thinking = false %} in the
Jinja chat template, or pass
extra_body={"chat_template_kwargs": {"enable_thinking": false}} on OpenAI-compatible
endpoints.
Part of Rogue Quants · NVFP4 component datasheets · collection. Fabricated on GB10 (Blackwell) with llm-compressor. Refusals shown per 100 harmful prompts; "n/a" = not separately measured (base-inherited).
