Credits
- Used mmangkad/Qwen3.6-27B-NVFP4 as base model (Thank you for doing ModelOpt quant!)
Notes
- Another mixed precision quant: ssm_out is bf16, other ssm norms and weights are in F32.
- Wv and attention output is in bf16.
- FFN tensors are in nvfp4, with modelopt's input scale / weight scale kept.
Currently llama.cpp doesn't have the best implementation for correctly derived nvfp4, but it's only a
small loss in perplexity.
- Resulting BPW is 6.98, for 5090 you can do 130k KV cache at 16 bit.
- Recommend FP16 KV cache. Do NOT follow previous recommendation for BF16 KV cache!!!
Flow
- convert_hf_to_gguf with mmangkad's nvfp4 checkpoint
- generate "keep" list to preserve tensors
- llama-quantize to q8_0 for the rest of the tensors
NVFP4
NVFP4 related tensors are untouched: weight scales for proper fp4 scaling are kept unchanged.
Image
For image, get any of the mainstream mmproj-bf16 gguf.
