Property	Value
Base model	Qwen3.6-35B-A3B
Source checkpoint	RedHatAI/Qwen3.6-35B-A3B-NVFP4
Quantization	NVFP4 (compressed-tensors)
Format	GGUF (21 GB, 1293 tensors)
Hidden size	2048
Layers	40 (Mixture-of-Experts)
Total experts	256
Experts per token	8
Expert FFN size	512
Shared expert FFN size	512
Context length	262,144
Vocabulary size	248,320
Tokenizer	Qwen3 (BPE, `qwen35` pre-tokenizer)

Architecture

Qwen3.6-35B-A3B is a mixture-of-experts multimodal model with:

Hybrid attention: alternating linear (Mamba-style SSM) and full attention layers every 4 blocks
256 experts with 8 activated per token (grouped-query MoE)
NVFP4 quantization via compressed-tensors on all Linear weight layers, preserving auxiliary tensors (norms, SSM convolutions, router gates) in BF16/F32
Vision encoder: Qwen2.5-VL-style ViT with 1152 hidden dim, merged into the language model

How This GGUF Was Created

Prerequisites

Clone llama.cpp with NVFP4 support:

bash
1git clone https://github.com/ggml-org/llama.cpp.git 2cd llama.cpp 3git checkout b8967

Version b8967 (or later) includes PR #21095 for converting HuggingFace NVFP4 models quantized with compressed-tensors, plus native Blackwell NVFP4 CUDA support via PR #22196. The converter maps compressed-tensors tensor names (e.g., weight_global_scale) to the ModelOpt equivalents and adjusts scale values to their reciprocals — weights themselves need no adaptation.
Install conversion dependencies (Python 3.12 or 3.13):

pip install -r requirements/requirements-convert_hf_to_gguf.txt --no-cache-dir

Conversion Steps

Download the source model:

huggingface-cli download RedHatAI/Qwen3.6-35B-A3B-NVFP4 --local-dir ./model
Apply tokenizer compatibility patch — the upstream tokenizer_config.json uses TokenizersBackend which is not recognized by the converter:

sed -i 's/"TokenizersBackend"/"Qwen2Tokenizer"/g' ./model/tokenizer_config.json
Register the BPE pre-tokenizer hash — the Qwen3.6 tokenizer produces an unrecognized hash in get_vocab_base_pre(). Add this entry to convert_hf_to_gguf.py:

python
1if chkhsh == "1444df51289cfa8063b96f0e62b1125440111bc79a52003ea14b6eac7016fd5f": 2 # ref: https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4 3 res = "qwen35"
Run the conversion:

bash
1python convert_hf_to_gguf.py ./model \ 2 --outfile Qwen3.6-35B-A3B-NVFP4.gguf \ 3 --outtype bf16 \ 4 --verbose

Important Notes

The --outtype bf16 flag does not produce a standard BF16 model. NVFP4-compatible tensors are repacked into the GGUF where supported by the converter and runtime. Auxiliary tensors (layer norms, SSM parameters, router gates) remain in floating-point format.
The BPE pre-tokenizer hash (1444df51...) was added to get_vocab_base_pre() mapping to the qwen35 tokenizer preset, matching the \p{M} (combining marks) regex used by Qwen3 models.

Benchmark Results

Benchmarks run using llama-bench from the llama.cpp Docker image (fc2b0053f / 8967) on CUDA 13 with a single GPU. Each model was tested at context lengths 2048, 4096, and 8192 tokens (prompt processing and 32-token token generation).

Qwen3.6-35B-A3B: NVFP4 vs Q4_K_XL

Quantization	Size	Params	PP 2048 t/s	PP 4096 t/s	PP 8192 t/s	TG 32 t/s
NVFP4	20.94 GB	34.66B	9767 ± 67	9572 ± 16	9107 ± 14	136 ± 19
Q4_K_XL	20.81 GB	34.66B	9121 ± 33	8889 ± 47	8464 ± 48	188 ± 23

Qwen3.6-27B: NVFP4 vs Q4_K_XL

Quantization	Size	Params	PP 2048 t/s	PP 4096 t/s	PP 8192 t/s	TG 32 t/s
NVFP4	17.50 GB	26.90B	5168 ± 28	5020 ± 4	4687 ± 6	63 ± 5
Q4_K_XL	16.39 GB	26.90B	3720 ± 10	3627 ± 6	3419 ± 13	66 ± 4

Conclusions

Prompt processing (PP): NVFP4 is significantly faster than Q4_K_XL across both model sizes. For 35B-A3B: ~7% at 2048 context (+8% at 4096, +7.5% at 8192). For 27B: ~39% faster consistently (~37-40% across contexts). This reflects NVFP4's lower computational overhead during attention computation since weights stay in their native FP4 format rather than being dequantized on-the-fly.
Token generation (TG): Q4_K_XL is ~28% faster at TG for the 35B-A3B (188 vs 136 t/s). For the 27B the gap is negligible — Q4_K_XL is only ~5% faster (66 vs 63 t/s). The NVFP4 path has higher per-token overhead in MoE routing and FFN computation, but this is less impactful at smaller scale.
Model size: For 35B-A3B, NVFP4 is marginally larger (20.94 vs 20.81 GB). For 27B, NVFP4 adds ~1.1 GB (17.50 vs 16.39 GB).
Recommendation: For 27B, NVFP4 is essentially a win — same TG throughput with significantly faster prompt processing. For 35B-A3B the choice depends on workload: NVFP4 for prompt-heavy use cases (RAG with long context windows), Q4_K_XL for interactive chat where per-token latency matters more.

Benchmark Methodology

podman run --rm --name llama-bench-<id> \
  --device=nvidia.com/gpu=all \
  --security-opt label=disable \
  -v ~/.cache/huggingface/:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full-cuda13-b8967 \
  --bench -hf <model> -n 3 --n-prompt <context> --n-gpu-layers 999

Full benchmark script: benchmark.sh

Multimodal Projector

This model includes a bundled multimodal projector (mmproj-BF16.gguf) for vision tasks. It is shared with the base Qwen3.6-35B-A3B architecture and sourced from Unsloth's GGUF conversion.

When using --hf, llama.cpp loads it automatically. For local usage, specify it with:

./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"

Usage

bash

# Local usage (specify mmproj for vision)
./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"

# Via HuggingFace (auto-discovers mmproj)
./llama-cli --hf knoopx/Qwen3.6-35B-A3B-NVFP4-GGUF -n 256 --prompt "Hello"

License

The original Qwen3.6 model is licensed under the Apache 2.0 License. This GGUF conversion is a format transformation of that checkpoint and inherits the same license. See RedHatAI/Qwen3.6-35B-A3B-NVFP4 for full licensing details.