Multi-Token Prediction (MTP) draft GGUF for NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, for use as a speculative decoding draft with llama.cpp's --spec-type draft-mtp.

The original NVIDIA release contains MTP (NextN) layers. This file restores the MTP head so llama.cpp users can get speculative decoding on this model.

mtp-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.gguf (4.1 GB, F16) — MTP draft model: the NextN block (dense attention sub-layer + 128-expert MoE sub-layer), token embeddings, final norm and LM head, exported with convert_hf_to_gguf.py --mtp. Standalone metadata: nemotron_h_moe, 53 blocks (52 + 1 NextN), 1M context.

Usage (llama.cpp, requires `nemotron_h_moe` support)

bash

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf \
  -hfrd mtp-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3

The main model can be any GGUF quant of this model (Q4_K_M, Q6_K, Q8_0, ...) — the draft head is applied on top of it. Tokenizer/vocab of the draft matches the base model.

Conversion

bash

git clone https://github.com/ggml-org/llama.cpp   # build with nemotron_h_moe support
python3 convert_hf_to_gguf.py <hf-dir> --mtp --outtype f16

Requires only the MTP-bearing safetensors shard (model-00014-of-00014.safetensors) plus the shards holding token embeddings, final norm and LM head; a filtered model.safetensors.index.json allows converting without the full 65 GB BF16 checkpoint.

License

Derived from NVIDIA weights — OpenMDW-1.1, see LICENSE (attribution: NVIDIA CORPORATION & AFFILIATES).

h1st0ry3D/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MTP-GGUF

MTP draft head for NVIDIA Nemotron 3.5 Lightning (GGUF)

Contents

Usage (llama.cpp, requires nemotron_h_moe support)

Conversion

License

h1st0ry3D/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MTP-GGUF

Usage (llama.cpp, requires `nemotron_h_moe` support)