A compressed-tensors NVFP4 W4A4 release of heretic-org/Qwen3.8-27B-heretic-ara with the original BF16 vision/video tower and all 15 native BF16 MTP tensors retained and runtime-validated on one RTX 5090.

Release artifact: this repository contains the converted nvfp4-pack-quantized checkpoint. The raw ModelOpt export was an intermediate artifact. Conversion to compressed-tensors is required for the tested Qwen3.8 multimodal + MTP vLLM path.

Quick start — RTX 5090 / Blackwell

Primary runtime: vLLM's Qwen3.8 CUDA 13 build. The packed NVFP4 weights, multimodal path, and native MTP drafter were validated with the pinned image below. Hugging Face may display an automatic 8-bit badge because two FP4 values are physically packed in each U8 storage element; the quantization is NVFP4 W4A4.

Download

bash

hf download lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --local-dir ./qwen38-nvfp4

Allow roughly 23 GB for the release plus temporary/cache headroom.

Recommended text-only long-context profile — 192K

This is the practical RTX 5090 profile: FP8 KV, one sequence, chunked prefill, prefix caching, native MTP n=3, FlashInfer/CUTLASS NVFP4 GEMM, and the vision graph disabled to preserve KV capacity.

bash

docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp \
  --host 0.0.0.0 --port 8000 \
  --language-model-only \
  --max-model-len 196608 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Aggressive text-only profile — 256K

Use the same command with:

text

--max-model-len 262144
--gpu-memory-utilization 0.96
--max-num-batched-tokens 8192

This is a single-concurrency capacity target. Add --enforce-eager when CUDA Graph capture or workspace overhead prevents startup; eager mode trades some performance for lower graph-memory pressure. Start with 192K for a stable operational profile.

Full image/video profile

The vision tower and encoder cache need additional VRAM. This pinned profile is the verified smoke-test configuration:

bash

docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 4096 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For multimodal capacity tuning, 32K is the conservative starting point, 64K–96K is the practical target range, and 128K is aggressive on 32 GB. Validate the chosen image/video limits and encoder-cache budget on your runtime before production.

OpenAI-compatible text request

bash

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"qwen38-nvfp4-mtp",
    "messages":[{"role":"user","content":"Explain speculative decoding briefly."}],
    "max_tokens":256,
    "chat_template_kwargs":{"enable_thinking":false}
  }'

Image request

python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="qwen38-nvfp4-mtp-vl",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
            {"type": "text", "text": "Describe this image."},
        ],
    }],
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(r.choices[0].message.content)

RTX 5090 optimization notes

Keep the GPU power limit at the operator-selected 400 W for sustained runs.
flashinfer-cutlass is the preferred NVFP4 GEMM backend on SM120; the validated runtime also logged FlashInferCutlassNvFp4LinearKernel.
FP8 KV is essential for 192K–262K capacity. BF16 KV roughly doubles KV memory.
max_num_seqs=1 reserves capacity for one very long conversation. Increase concurrency only after lowering context or measuring free KV blocks.
Chunked prefill controls peak scheduling pressure for long prompts; prefix caching accelerates repeated prefixes.
expandable_segments:True reduces allocator fragmentation during long-context and multimodal workloads.
Keep MTP at n=3 as the validated speed/acceptance point. Workload-specific acceptance determines the actual gain.
The pinned digest is the compatibility contract. A newer vLLM tag should be revalidated for Qwen3.8, compressed-tensors NVFP4, vision/video, MTP, tool parser, and long-context allocation.
The model card's throughput values come from 4K validation runs. 192K/262K settings are capacity profiles; long-prompt prefill throughput is workload-dependent.

Lineage

Component	Source
Official architecture	Qwen/Qwen3.8-27B
Quantized source	heretic-org/Qwen3.8-27B-heretic-ara @ `2dc9b364104881cbb85e390f00195ba6b9d745e9`
Behavioral lineage	Heretic / ARA / abliterated
Vision/video tower	333 tensors retained from the same upstream checkpoint, BF16
MTP head	15 tensors retained from the same upstream checkpoint, BF16
PTQ	NVIDIA Model Optimizer NVFP4 W4A4
Portable packaging	compressed-tensors `ModelOptNvfp4Converter`

Heretic v1.2.0+custom ARA de-censoring of Qwen/Qwen3.8-27B, published by trohrbaugh and mirrored under heretic-org. The two source repositories were checked and their model artifacts are byte-identical; the organization mirror adds base-model metadata.

What is quantized

Component	Format
Language-model Linear layers	NVFP4 W4A4, group size 16
Vision/video tower (`model.visual.*`, 333 tensors)	BF16
MTP (`mtp.*`, 15 tensors)	BF16
`lm_head` and token embeddings	BF16
Gated DeltaNet `conv1d`	BF16
Norms, biases and small state tensors	BF16/FP32 as exported

Checkpoint metadata:

text

quant_method: compressed-tensors
format: nvfp4-pack-quantized
tensors: 2687
weight files: ~20.56 GB
calibration: 20 samples × 8192 tokens = 163,840 tokens
calibration dataset: abisee/cnn_dailymail, config 3.0.0

Validation

Hardware and runtime:

text

GPU: NVIDIA GeForce RTX 5090 32 GB (SM120)
GPU power limit: 400 W
Runtime: vllm/vllm-openai:qwen38-x86_64-cu130
Runtime digest: sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e
vLLM build: 0.1.dev19754+g3a0914114
PyTorch: 2.13.0+cu130
KV cache: FP8
MTP: n=3

Verified paths:

GET /health → HTTP 200
GET /v1/models → expected served model
OpenAI-compatible text request → HTTP 200
Real PNG request → HTTP 200: A red square and a blue circle are displayed on a white background.
Real MP4 request → HTTP 200: The video shows a red square and a blue circle on a white background.
Native MTP drafter loaded; embeddings and LM head shared with target model

Measured single-stream results from 1024-token generations:

Metric	Result
Client-observed generation throughput with MTP	120.5–134.7 tok/s
Mean MTP acceptance length	2.83–2.86
Per-position acceptance	`0.777/0.592/0.463 and 0.808/0.599/0.456`
Average draft acceptance	61.1–62.1%
Runtime VRAM during multimodal request	~28,984 MiB

These are local measurements for the stated prompts/configuration. They are reproducibility evidence, not broad quality benchmarks or a guaranteed speedup over every non-MTP setup.

Compact reference command

bash

docker run --rm --gpus all --ipc=host --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:qwen38-x86_64-cu130 \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' 

For short non-thinking responses, pass this at the top level of the request:

{"chat_template_kwargs": {"enable_thinking": false}}

Use the pinned profiles in Quick start for production. The 192K text-only profile is the recommended RTX 5090 long-context setup; 262K is the aggressive single-sequence target. The compact command above remains a readable 32K multimodal example.

Reproduction outline

Load the full BF16 VLM with Qwen3_5ForConditionalGeneration using CPU/disk offload.
Quantize the language model with ModelOpt NVFP4_DEFAULT_CFG.
Calibrate on 20 CNN/DailyMail samples at sequence length 8192.
Keep lm_head, token embeddings, linear_attn.conv1d, visual.*, and mtp.* outside NVFP4.
Export the ModelOpt HF checkpoint.
Retain/graft the 15 MTP tensors from the same upstream source.
Convert ModelOpt tensor conventions to compressed-tensors using ModelOptNvfp4Converter:
- weight → weight_packed
- input_scale → reciprocal input_global_scale
- weight_scale_2 → reciprocal weight_global_scale
Validate image, MP4, text, and MTP n=3 in the exact Qwen3.8 runtime above.

The converter excludes:

text

lm_head
model.language_model.embed_tokens
re:.*visual.*
re:.*conv1d.*
re:^mtp.*

See BUILD_MANIFEST.json, VALIDATION_REPORT.json, recipe.yaml, and SHA256SUMS in this repository.

Sources and acknowledgements

Qwen/Qwen3.8-27B — official architecture, tokenizer, vision/video stack, and trained MTP head
NVIDIA Model Optimizer — initial NVFP4 W4A4 PTQ/export
compressed-tensors — ModelOptNvfp4Converter and portable nvfp4-pack-quantized packaging
vLLM — Qwen3.8 serving, multimodal API, and MTP speculative decoding
sakamakismile/Qwen3.8-27B-MTP-NVFP4 — working Qwen3.8 compressed-tensors NVFP4+MTP runtime reference
sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP — earlier ModelOpt NVFP4+MTP recipe reference
CNN/DailyMail 3.0.0 — calibration text source
trohrbaugh/Qwen3.8-27B-heretic-ara — original publisher path for this ARA artifact
Heretic, timrohrbaugh/heretic, and ARA PR #211 — de-censoring method/tool lineage

All upstream behavioral changes belong to their respective source authors. This repository contributes quantization, packaging, and RTX 5090 runtime validation. It does not claim authorship of Qwen3.8, Heretic/ARA, or Blackfrost's model modifications.

Limitations

NVFP4 requires compatible NVIDIA Blackwell kernels/runtime.
The tested full multimodal profile uses most of a 32 GB RTX 5090.
FP8 KV scale warnings may appear when static q/prob scales are absent; the tested runtime uses fallback scale 1.0.
The calibration set is compact. Task-specific quality should be evaluated against the BF16 source before production use.
Abliterated/de-censored behavior comes from the upstream model. Operators are responsible for deployment policy and outputs.

License

Apache-2.0, following Qwen3.8 and the cited upstream checkpoint. The included LICENSE is copied from the upstream release.