NVIDIA NVFP4 quantized version of Mixtral 8x7B-Instruct for Blackwell architecture GPUs.
Model Description
This is a 4-bit floating-point (NVFP4) quantized version of mistralai/Mixtral-8x7B-Instruct-v0.1, created using NVIDIA TensorRT Model Optimizer (modelopt).
Metric
Value
Original Size
86.99 GB
Quantized Size
24.82 GB
Compression Ratio
3.50x
Size Reduction
71.5%
Quantization Method
NVFP4 (calibration-based)
Calibration Samples
512 (C4 dataset)
Quantization Time
~5.6 hours
What is NVFP4?
NVFP4 is NVIDIA's native 4-bit floating-point format, introduced with the Blackwell architecture. Unlike integer quantization (INT4), NVFP4 uses a micro-exponent floating-point format:
┌───────┬───────────┬──────────┐
│ Sign │ Exponent │ Mantissa │
│ 1 bit │ 2 bits │ 1 bit │
└───────┴───────────┴──────────┘
This provides better dynamic range for neural network weights compared to uniform integer quantization.
Hardware Requirements
⚠️ This model requires Blackwell architecture GPUs (GB10, GB100, GB200) and TensorRT-LLM for inference.
Standard HuggingFace transformers cannot load this model directly due to the packed FP4 weight format.
Current Compatibility Status (December 2025)
Framework
Status
TensorRT-LLM
Partial support (GB10 not fully supported in v1.0.0)
vLLM
Not yet supported
Transformers
❌ Cannot load packed FP4 weights
This model is ready for when TensorRT-LLM and vLLM add full Blackwell support.
All block_sparse_moe.gate layers (router networks)
Calibration
Quantization was performed with 512 calibration samples from the C4 dataset, running forward passes to collect weight and activation statistics for optimal scale factor determination.
Baseline Performance (BF16)
For reference, the original BF16 model on DGX Spark (GB10):
