This quantization reduces memory from 87 GB to 25 GB (72% reduction) while improving MMLU accuracy by +2.5 points over the BF16 baseline. Inference throughput is 2.55x faster.

Key Results

Metric	NVFP4	BF16	Delta
MMLU Overall	60.0%	57.5%	+2.5%
Wikitext Perplexity	7.36	6.20	+1.17
Inference Speed	20.7 tok/s	8.1 tok/s	2.55x
Model Size	25 GB	87 GB	-72%

NVFP4 preserves and slightly improves task-level reasoning (MMLU) while trading off raw language modeling fidelity (perplexity). The mechanism behind the MMLU improvement is not fully understood.

MMLU Breakdown (Full 14K+ Questions, 5-shot)

Category	NVFP4	BF16	Delta
Overall	60.0%	57.5%	+2.5%
STEM	51.7%	49.4%	+2.3%
Humanities	54.5%	50.6%	+3.9%
Social Sciences	68.5%	67.5%	+1.0%
Other	68.4%	66.3%	+2.1%

Both models were benchmarked under identical conditions: same hardware, same vLLM version, same context window (4096 tokens), same evaluation harness.

Wikitext Perplexity

Metric	NVFP4	BF16
Word Perplexity	7.364	6.196
Bits per Byte	0.539	0.492
Byte Perplexity	1.453	1.406

BF16 retains an advantage on raw next-token prediction, as expected for an unquantized model.

Quantization Details

Parameter	Value
Algorithm	NVFP4 E2M1, max calibration
Tool	NVIDIA ModelOpt 0.41.0 (`NVFP4_DEFAULT_CFG`)
Group Size	16
Calibration Data	512 samples (256 from `codeparrot/codeparrot-clean`, 256 from `Open-Orca/SlimOrca`)
Calibration Batch Size	8, pre-tokenized, max_length=512
Quantization Time	~1 hour
Excluded Layers	`lm_head`, all `self_attn` layers, all `block_sparse_moe.gate` (router) layers
Export Format	HuggingFace safetensors via `modelopt.torch.export.export_hf_checkpoint`

Why Exclude Self-Attention?

An initial quantization (v1) applied NVFP4 to all linear layers except embeddings and routers, yielding 57.0% MMLU — a 2.9% drop from BF16 baseline (measured with --limit 200). Excluding self-attention layers from quantization (v2) recovered that loss and pushed accuracy above baseline to 61.1% on the same subset and 60.0% on the full 14K evaluation. The MoE expert layers quantize well; attention layers do not.

Benchmark Methodology

All benchmarks used lm-evaluation-harness v0.4.11.

Parameter	Value
MMLU Task	`mmlu_generative` (57 subjects, generative format)
Few-shot	5-shot
Wikitext Task	`wikitext` (perplexity via `local-completions`)
Context Window	4096 tokens (both models)
Concurrency	5 concurrent requests
Hardware	NVIDIA DGX Spark (GB10), 128 GB unified memory
Inference Stack	vLLM 0.14.0rc1 on `gogamza/unsloth-vllm-gb10:latest`

Preliminary vs Full Benchmark

An initial benchmark with --limit 200 (200 questions per subject) showed NVFP4 at 61.1% vs BF16 at 59.9% (+1.2%). The full benchmark (14K+ questions, no limit) confirmed and widened this result to 60.0% vs 57.5% (+2.5%), providing stronger statistical confidence.

Inference Speed

Measured separately: 5 runs of 500-token generation, single concurrent request, technical prompt.

Model	Tokens/sec	Relative
NVFP4	20.7 tok/s	2.55x
BF16	8.1 tok/s	1.0x

Hardware Requirements

GPU: NVIDIA Blackwell (SM100+) — GB10, GB100, GB200
Memory: ~25 GB for model weights + KV cache overhead
Tested on: DGX Spark (GB10), 128 GB unified LPDDR5x

This model will not run on pre-Blackwell GPUs (A100, H100, etc.). NVFP4 is a Blackwell-native format.

Usage

vLLM

bash

docker run -d --name mixtral-nvfp4 \
    --runtime nvidia --gpus all \
    -p 8001:8001 \
    -v /path/to/Mixtral-8x7B-Instruct-v0.1-NVFP4:/model \
    -v /path/to/cache:/root/.cache/vllm \
    gogamza/unsloth-vllm-gb10:latest \
    vllm serve /model \
    --host 0.0.0.0 --port 8001 \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.3 \
    --served-model-name mixtral \
    --chat-template /model/chat_template.jinja

vLLM auto-detects NVFP4 quantization from hf_quant_config.json. No --quantization flag needed.

OpenAI-Compatible API

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="none")
response = client.chat.completions.create(
    model="mixtral",
    messages=[{"role": "user", "content": "Explain the Mixture of Experts architecture."}],
)
print(response.choices[0].message.content)

Files

File	Description
`model-*.safetensors` (6 shards)	NVFP4 quantized weights (~25 GB total)
`config.json`	Model config with quantization parameters
`hf_quant_config.json`	ModelOpt quantization metadata (auto-detected by vLLM)
`tokenizer.json`, `tokenizer_config.json`	Tokenizer files
`chat_template.jinja`	Chat template for instruct format
`special_tokens_map.json`	Special token definitions

Known Limitations

Blackwell-only: Requires SM100+ hardware. Will not load on Ampere/Hopper.
Perplexity trade-off: ~19% higher word perplexity vs BF16. Raw language modeling quality is reduced, though task performance is preserved.
No KV cache quantization: Only weights and activations are quantized. KV cache remains in default precision.
HumanEval: Not benchmarked — lm-eval's generative evaluation mode conflicts with chat template wrapping, producing 0.0 pass@1 for both NVFP4 and BF16.

License

Apache 2.0

Citation

bibtex

@misc{brewhaha2026mixtral-nvfp4,
    title={Mixtral-8x7B-Instruct-v0.1-NVFP4: NVFP4 Quantization with Selective Layer Exclusion},
    author={SyntacticallySugary},
    year={2026},
    url={https://huggingface.co/SyntacticallySugary/Mixtral-8x7B-Instruct-v0.1-NVFP4}
}

Acknowledgments

Mistral AI for the base Mixtral-8x7B-Instruct model
NVIDIA for ModelOpt, NVFP4 format, and the DGX Spark platform
vLLM for inference serving with Blackwell NVFP4 support
EleutherAI for the evaluation harness