
Source	`Qwen/Qwen3-4B`
Scheme	`NVFP4` (4-bit)
Format	compressed-tensors
Parameters	4.0B
Size on disk	2.8 GB
Compression	2.84x smaller than the 8.0 GB source
Calibration	`HuggingFaceH4/ultrachat_200k`, 256 samples
Left unquantized	`lm_head`
Quantized on	A40
Quantized by	Sohailhosseini

Usage

bash

vllm serve Sohailhosseini/Qwen3-4B-NVFP4 \
  --max-model-len 32768

python

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/Qwen3-4B-NVFP4", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)

Measured quality

Downloaded from this repo and run on transformers 5.15.1 / compressed-tensors 0.18.0 (CPU, greedy decoding). It loads and generates coherently - so what is published here is intact, not merely what was built.

Not yet verified under vLLM. NVFP4 inference needs FlashInfer's FP4 kernels, and on every Blackwell card obtainable at the time of writing FlashInfer reported requires GPUs with sm75 or higher on an sm_120 device - a detection problem rather than a real capability shortfall, but one that prevents the server from starting. The vllm serve command above is therefore the documented usage, not a tested one. The sibling FP8 and AWQ-W4A16 quants of other models in this collection have been served and API-tested; this one has not.

Provenance

Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers, calibration set and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.