The original upstream checkpoint is multimodal. This repo contains:
the text-only language model extracted from the original checkpoint
quantized to full NVFP4
saved in compressed-tensors format
adjusted for vLLM loading
I'm running this with about 21 token/s per request in vllm using the openai style api on a GB10 based machine.
The Hugging Face UI may report unusual parameter and tensor-type information for this model. That is expected for compressed NVFP4 artifacts and should not be interpreted as the original dense model having changed to a literal 5B FP32/BF16 model.
Important notes
This is not the original upstream model.
This is not multimodal. Vision/image support was removed during extraction.
