Two-level scaling: E4M3 FP8 scaling per 16-value block + global FP32 tensor scale
Hardware acceleration: Optimized for Tensor Cores on Blackwell GB10 GPUs
Group size: 16
Minimal accuracy degradation: Less than 1% vs original model
Excluded modules: lm_head (kept in higher precision)

Calibration

Dataset: C4 (Colossal Clean Crawled Corpus)
Calibration samples: 512
Maximum sequence length: 2048 tokens
Method: Post-training quantization with activation calibration

Usage

Requirements

NVIDIA DGX Spark or compatible Blackwell GPU
vLLM >= 0.6.5
nvidia-modelopt[hf]

Loading the Model

IMPORTANT: This model must be loaded with vLLM using the modelopt quantization parameter. Standard HuggingFace AutoModelForCausalLM will not work.

python

from vllm import LLM, SamplingParams

# Load NVFP4 quantized model
llm = LLM(
    model="Ex0bit/Qwen3-VLTO-32B-Instruct-NVFP4",
    quantization="modelopt",  # Required for NVFP4
    trust_remote_code=True,
    gpu_memory_utilization=0.9
)

# Generate
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)
outputs = llm.generate(["Explain quantum computing in simple terms:"], sampling_params)
print(outputs[0].outputs[0].text)

Environment Variables

You can optionally set:

HF_CACHE_DIR: Override HuggingFace cache location

Limitations

Hardware specific: Optimized for NVIDIA Blackwell architecture (GB10)
vLLM required: Cannot be loaded with standard transformers library
Quantization artifacts: Minor precision loss (<1%) compared to BF16 original

Intended Use

This model is intended for:

High-throughput inference on NVIDIA DGX Spark systems
Production deployments requiring memory-efficient models
Research on quantization techniques for large language models

Training and Quantization

Base Model Training

See the original model card for base model training details.

Quantization Process

Model Loading: Original model loaded in BF16 precision
Calibration: 512 samples from C4 dataset for activation statistics
Quantization: NVFP4 format applied using NVIDIA modelopt
Export: Saved in HuggingFace safetensors format

Quantization Time: Approximately 60-90 minutes on DGX Spark

Evaluation

Test Results

All 5 inference tests passed successfully:

Technical explanation generation
Code generation
Mathematical reasoning
Creative writing
Instruction following

Average performance: 9.99 tokens/s on DGX Spark GB10

Citation

If you use this quantized model, please cite:

bibtex

@misc{qwen3vlto32b-nvfp4,
  author = {Ex0bit},
  title = {Qwen3-VLTO-32B-Instruct-NVFP4: NVFP4 Quantized Model for DGX Spark},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Ex0bit/Qwen3-VLTO-32B-Instruct-NVFP4}},
}

And the original base model:

bibtex

@misc{qingy2024qwen3vlto,
  author = {qingy2024},
  title = {Qwen3-VLTO-32B-Instruct},
  year = {2024},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/qingy2024/Qwen3-VLTO-32B-Instruct}},
}

References

License

This quantized model inherits the license from the base model. Please refer to the original model's license for details.

Model Card Authors

Ex0bit (@Ex0bit)

Acknowledgments

NVIDIA for TensorRT Model Optimizer and DGX Spark hardware
qingy2024 for the base Qwen3-VLTO-32B-Instruct model
The vLLM team for high-performance inference framework