NVFP4 (native Blackwell FP4) GGUF of Qwen3.6-35B-A3B (Mixture-of-Experts, ~3B active parameters per token). A single-file, FP4-native quantization that runs on LocalAI's paged-attention llama.cpp backend with strong decode throughput and a low memory footprint on Blackwell-class GPUs (GB10 / DGX Spark).

Blackwell GPU recommended (native FP4-MMA). It also runs on other hardware via NVFP4 dequant, but slower - off Blackwell the FP4 matmuls fall back (e.g. to CPU on Apple Metal). The throughput figures below are measured on GB10 / DGX Spark (consumer Blackwell).

File

File	Size	file_type	NVFP4 tensors	sha256
`q36-35b-a3b-nvfp4.gguf`	23.9 GB (23,850,227,680 bytes)	`MOSTLY_NVFP4`	241	`1690d0424e232527b8bb135a38033e4699ad11817677eebacd40349020faea52`

Source / attribution

Weights: MoE NVFP4 weights from nvidia/Qwen3.6-35B-A3B-NVFP4.
Conversion: converted to GGUF with llama.cpp using --outtype auto (NVFP4-preserving). The NVFP4 tensors are carried through into the GGUF unchanged, so the file reports file_type = MOSTLY_NVFP4 with 241 native NVFP4 tensors. No re-quantization to an integer K-quant is performed.

base_model: nvidia/Qwen3.6-35B-A3B-NVFP4

License

Released under the Qwen license (see https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE). This repository redistributes the converted GGUF with attribution to the upstream weights; the original model license and terms apply.

Install in LocalAI

This model is pre-configured in the LocalAI model gallery for the paged-attention backend - install it by name and it is ready to serve, with no manual configuration.

Gallery model name: qwen3.6-35b-a3b-nvfp4-paged

Heads-up - the backend is not in a release yet. The llama-cpp-localai-paged backend and these gallery entries ship in LocalAI PR #10462, which is not yet merged. Until it lands, run LocalAI built from that PR branch. Once it is merged, a current LocalAI release already includes the backend and this gallery entry - nothing extra needed.

Install from the CLI:

local-ai models install qwen3.6-35b-a3b-nvfp4-paged

Or from the WebUI: open the Models gallery, search qwen3.6-35b-a3b-nvfp4-paged, and click Install.

Then call it through the OpenAI-compatible API:

bash

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen3.6-35b-a3b-nvfp4-paged",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Pre-configured serving config

The gallery entry already applies the optimized paged setup the benchmarks below use - no tuning required:

Paged KV cache (on-demand, paged-attention backend)
Decode-first prefill budget (prefill scheduled to keep decode latency low)
Flash attention
Optional bf16-SSM fast mode via the ssm_bf16_tau option (see Benchmarks)

LocalAI: https://github.com/mudler/LocalAI

Benchmarks

Measured on GB10 / DGX Spark with the LocalAI paged-attention setup:

Decode throughput reaches ~86% of vLLM at 128-way concurrency (bit-exact, MoE active-expert path).
1.5-3x lower memory than the comparable vLLM deployment.
At equal-or-higher precision (NVFP4 native execution).
The opt-in bf16-SSM fast mode (ssm_bf16_tau) closes most of the remaining gap.

Verification

sha256sum q36-35b-a3b-nvfp4.gguf
# 1690d0424e232527b8bb135a38033e4699ad11817677eebacd40349020faea52

SHA256SUMS.txt (covering both NVFP4 GGUFs in this collection) is included in the repo.

mudler/Qwen3.6-35B-A3B-NVFP4-GGUF

Qwen3.6-35B-A3B NVFP4 GGUF