Qwen2.5-7B-Instruct-NVFP4A16
NVFP4A16 quantization of
Qwen/Qwen2.5-7B-Instruct, produced with
llm-compressor and
published only after passing an automated accuracy gate against the bf16 baseline (see
Certification).
Model overview
- Scheme:
NVFP4A16 - 4-bit NVFP4 weights, 16-bit (bf16) activations (weight-only).
- Why weight-only: activations are transient (never stored), so keeping them at 16-bit costs ~nothing on disk - weights dominate size. Fully 4-bit activation quantization (W4A4) is widely reported to cost token-level quality; we have not measured that ourselves, and the one W4A4 candidate assay did gate failed the perplexity bar at +12.55% and was rejected. You get almost all the compression at a quality cost this card measures rather than estimates.
- Format: compressed-tensors; loads natively in vLLM (no
--quantization flag).
Use with vLLM
1from vllm import LLM, SamplingParams
2
3llm = LLM(model="uist-labs/Qwen2.5-7B-Instruct-NVFP4A16")
4prompts = ["Give me a short introduction to large language models."]
5params = SamplingParams(max_tokens=256)
6for out in llm.generate(prompts, params):
7 print(out.outputs[0].text)
Hardware requirements
This checkpoint uses weight-only NVFP4: it runs via vLLM's FP4 Marlin kernel and does not require Blackwell GPUs or native FP4 tensor cores. On non-Blackwell hardware the 4-bit weights are dequantized to 16-bit for the GEMM, trading some compute throughput for the smaller memory footprint (the KV cache and activations are 16-bit either way, so the saving is on the linear weights).
Validated (measured through assay's gate): Ada (sm_89: e.g. L4, RTX 4090). Other >= sm_80 (Ampere / Hopper) GPUs are expected to work by the same weight-only Marlin path but are not yet independently validated here. Turing (sm_75) is excluded: a known vLLM issue makes the NVFP4 Marlin GEMM emit incorrect output on Turing, so we do not claim it until that is fixed and measured.
Creation
- Base snapshot:
Qwen/Qwen2.5-7B-Instruct @ a09a35458c70 - upstream repos are mutable, so the certificate names the exact commit it describes; the staged weights were verified against the recipe's pinned sha256s before quantization.
- Tool: llm-compressor, scheme
NVFP4A16, targets Linear, ignore lm_head.
- Calibration: 512 samples from
HuggingFaceH4/ultrachat_200k at 2048-token sequences.
- Pipeline: UIST Labs'
assay benchmark-gated quantization pipeline (https://github.com/uist-labs/assay/tree/v0.6.2) - quantize -> benchmark -> gate -> publish.
Evaluation
Measured with lm-evaluation-harness (vLLM backend) on the bf16 baseline and this checkpoint. retention is quantized / baseline (higher is better; for perplexity, lower raw value is better).
| task | metric | baseline | quantized | delta | retention |
|---|
| gsm8k | exact_match,flexible-extract | 0.7127 | 0.7248 | +0.0121 | 1.0170 |
| arc_challenge | acc,none | 0.4531 | 0.4599 | +0.0068 | 1.0151 |
| hellaswag | acc,none | 0.5704 | 0.5713 | +0.0009 | 1.0016 |
| winogrande | acc,none | 0.6014 | 0.6085 | +0.0071 | 1.0118 |
| mmlu | acc,none | 0.6877 | 0.6891 | +0.0014 | 1.0021 |
| wikitext | word_perplexity | 9.4022 | 9.5615 | +0.1592 | 1.0169 |
Mean accuracy retention: 1.0095 - gate: PASS
Measurement environment
The evaluation numbers above were captured in the following environment.
| field | value |
|---|
| image | ghcr.io/uist-labs/assay@sha256:d06eceee8b70565f8b5eeafddaf6d0a519b35e8977a45569131076fddf3aba82 |
| GPU | NVIDIA GeForce RTX 5090, 32607 MiB VRAM, ECC not supported |
| driver / CUDA | 580.126.16 / 13.0 |
| torch | 2.11.0+cu129 |
| vllm | 0.25.0+cu129 |
| llmcompressor | 0.12.0 |
| compressed-tensors | 0.17.1 |
| lm_eval | 0.4.12 |
This measurement ran on hardware without memory-error protection (no ECC).
Methodology and limitations
- Apples-to-apples deltas. The bf16 baseline and this checkpoint were evaluated with the identical harness and settings, so the
delta/retention columns are a fair like-for-like comparison - which is what a quantization gate should measure: change from the original, honestly.
- Chat-mode evaluation. Tasks are evaluated with the model's chat template applied. The template shifts the absolute scores on both the baseline and the quantized model, so read the deltas, not the absolute values. The comparison stays valid because both sides are evaluated with identical settings.
- Scope: the certified quantity is the delta, not the absolute score. Both sides were measured in the same run against the same software stack, which is the only condition under which these two columns are comparable. Absolute scores can move further across harness or library versions, so a number here will not necessarily reproduce elsewhere, while the delta under identical conditions is what was certified. For the same reason we do not compare against a stored baseline from an earlier run: that would fold stack drift into the measurement.
- Retention near or above 100% means "no measurable loss," not "better." Where a task ticks up, that is sampling noise - a finite benchmark set is a sample, not the whole population - so read the whole table as "indistinguishable from the original," not as an improvement.
- Weight-only tradeoff. Weights are 4-bit; activations stay 16-bit. That keeps quality close to the original at nearly the full disk-size saving, at some inference-speed cost versus a fully 4-bit (W4A4) variant. We have not benchmarked a W4A4 build of this model, so we make no claim about where it lands on that curve.
- Bias, risks, and inherited behavior. This is a quantization of
Qwen/Qwen2.5-7B-Instruct and inherits its capabilities, biases, and limitations. The gate above certifies accuracy retention on the listed benchmarks only; it does not measure bias, safety, or any behavior those benchmarks do not cover, and quantization is not guaranteed to preserve what was not measured. For intended use, safety, and ethical considerations, refer to the base model's card.
Certification
UIST Labs publishes a quantized checkpoint only if it clears a hard, stated accuracy bar against its own bf16 baseline - we would rather withhold a release than ship an unverified one. This checkpoint passed all of:
- Mean accuracy retention >= 99%
- No single accuracy task down more than 2.0 points
- Perplexity increase <= 3%
The deltas above are the actual measured numbers, not vendor estimates. This gate runs on every UIST Labs quantization release.
Citation
If you use this checkpoint, please cite both this quantized release and the base model.
1@misc{uist_labs_qwen2_5_7b_instruct_nvfp4a16,
2 title = {Qwen2.5-7B-Instruct-NVFP4A16: benchmark-gated NVFP4 quantization of Qwen2.5-7B-Instruct},
3 author = {{UIST Labs}},
4 year = {2026},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/uist-labs/Qwen2.5-7B-Instruct-NVFP4A16}},
7 note = {Quantized and certified with the assay pipeline (https://github.com/uist-labs/assay/tree/v0.6.2); published only after passing an automated accuracy gate against the bf16 baseline.}
8}
Please also cite the base model,
Qwen/Qwen2.5-7B-Instruct.
Produced by assay v0.6.1 - UIST Labs
