MRockatansky/

NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-MTP-GGUF

GGUFs made with a non-mainline build of llama.cpp: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 (official), converted with convert_hf_to_gguf.py at Premshay/llama.cpp@731368b5d (branch nemotron-super-mtp, stacked on PR #25444), quantized with llama-quantize from the same build

The GGUFs should be loaded with the following fork of llama.cpp on the nemotron-super-mtp branch: https://github.com/Premshay/llama.cpp/tree/nemotron-super-mtp

Just an experiment to try the forked llama.cpp built on top of PR#25444 to support this model's built in MTP heads. Use the fork of llama.cpp above with --spec-type draft-mtp enabled in llama-server to provide a nice speedup in token generation.

The contributor of the fork explained their methodology on the model card below. I just did the same but with Nemotron Puzzle BF16 weights instead of using Nemotron Super: https://huggingface.co/Premshay/Nemotron-3-Super-120B-A12B-MTP-GGUF