This model was converted with the still-unmerged ggml-org/llama.cpp PR #25444, pinned to commit af49ef5cd990d039dbf360dd3a9f3b5dafdd1726, plus a narrowly scoped converter compatibility fix for the official BF16 checkpoint's model.layers.* tensor prefix and bounded writeback for large lazy tensors on the high-RAM Colab runtime. Until equivalent support is merged into mainline llama.cpp, use a build containing PR #25444 to load these files.

PR #25444 adds NemotronHPuzzleForCausalLM / nemotron_h_puzzle support, heterogeneous per-layer MoE settings, and the model's two-block MTP draft head. This repository is not an official NVIDIA or llama.cpp release.

Files

NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16.gguf — BF16 master GGUF
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q2_K.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q3_K_S.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q3_K_M.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q3_K_L.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-IQ4_XS.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q4_K_S.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q4_K_M.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q5_K_S.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q5_K_M.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q6_K.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q8_0.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q4_0.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q4_1.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q5_0.gguf
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-Q5_1.gguf

No importance matrix was used for any quantization in this repository.

Shape-fallback warning

Puzzle uses heterogeneous expert widths from 1280 to 2688. Some expert down-projection rows, especially width 2688, are not divisible by the 256-element block size used by K-quants. The pinned llama.cpp quantizer therefore applies its compatible fallback automatically on those tensors: Q2_K/Q3_K to Q4_0, IQ4_XS to IQ4_NL, Q4_K to Q5_0, Q5_K to Q5_1, and Q6_K to Q8_0. Consequently, named K-quant files are mixed recipes rather than a single tensor type everywhere. Required norms, scales, and other small tensors may remain F32.

Artifact integrity and quantizer compatibility are checked automatically, but successful conversion does not prove task-quality parity with BF16. Evaluate very small formats before practical use.

Provenance

Source revision: 8fe5546888e9bd03fdbf52d808adebdfca901b52
Source safetensors: 156,596,801,168 bytes across 31 model shards plus mtp.safetensors
llama.cpp PR: #25444
PR commit: af49ef5cd990d039dbf360dd3a9f3b5dafdd1726
Patched converter SHA-256: 1aaa36ac789fc6eceebefe19d4d80c3c9dc56185a4a3e956411bc0478ee46afc
Patched large-write GGUF writer SHA-256: 350db0132703b3b4025ee61e344b7d7400b9c3a2692b87dd6ca186d82423fa30
Conversion: pinned safetensors in hybrid RAM/disk storage, then local convert_hf_to_gguf.py --outtype bf16
Quantization: pinned llama-quantize, without --imatrix
License: the source model is governed by the OpenMDW License Agreement 1.1

Follow the prompting, safety, license, and usage guidance in the original NVIDIA model card.

RemySkye/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-GGUF

NVIDIA Nemotron Labs 3 Puzzle 75B-A9B GGUF

Important llama.cpp compatibility notice

Files

Shape-fallback warning

Provenance

RemySkye/
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-GGUF