You can deploy this model and start earning money today!

impacte/
NVIDIA-Nemotron-Nano-9B-v2-GGUF

Name: NVIDIA-Nemotron-Nano-9B-v2-GGUF
Availability: InStock

nvidia/NVIDIA-Nemotron-Nano-9B-v2

quantized

other

conversational

Views

No views yet

NVIDIA-Nemotron-Nano-9B-v2 (GGUF)

NVIDIA

Built by impacte.tech

Ollama

GGUF

GGUF conversions of nvidia/NVIDIA-Nemotron-Nano-9B-v2, a general-purpose instruction-tuned reasoning model from NVIDIA's Nemotron family, converted for use with llama.cpp.

Two quantizations are provided:

Quantization	File	Size	Use case
bf16	`NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf`	17.79 GB	Full precision — maximum quality, needs 24 GB+ VRAM
Q4_K_M	`NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf`	6.53 GB	Fits entirely on a single 16 GB GPU with ~9 GB left for KV cache

Model Summary

Property	Value
Base model	`nvidia/NVIDIA-Nemotron-Nano-9B-v2`
Architecture	`NemotronHForCausalLM` (`nemotron_h`)
Parameters	~9B
Layers	56
Attention heads	40 (8 KV heads)
Hidden size	4480
Context length	131,072
Vocab size	131,072
Reasoning	Yes (thinking mode, `/think` & `/no_think` controls)
Tool calling	Native (`<tool_call>` XML format)

Files

impacte/NVIDIA-Nemotron-Nano-9B-v2-GGUF/
├── NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf     # bf16 GGUF (17.79 GB)
├── NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf   # Q4_K_M GGUF (6.53 GB)
└── .gitattributes                            # LFS tracking

Usage

llama.cpp (local inference)

bash

# bf16 (full quality, 24 GB+ VRAM)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf \
  --ctx-size 8192 \
  --port 8080

# Q4_K_M (single 16 GB GPU)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf \
  --ctx-size 32768 \
  --port 8080

Then call the OpenAI-compatible endpoint:

bash

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M",
    "messages": [
      {"role": "user", "content": "Explain what a Tauri v2 app is."}
    ]
  }'

llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,  # offload all layers to GPU
)

About the base model

NVIDIA-Nemotron-Nano-9B-v2 is a compact, instruction-tuned reasoning model in NVIDIA's Nemotron family, trained on NVIDIA's post-training and pretraining datasets. It supports multiple languages (en, es, fr, de, it, ja) and is designed for efficient local deployment with a hybrid Mamba-2 + attention architecture.

License & Attribution

Base model: nvidia/NVIDIA-Nemotron-Nano-9B-v2 — released under the NVIDIA Open Model License. Review before commercial use.
GGUF conversion: performed with llama.cpp's convert_hf_to_gguf.py (bf16) and llama-quantize (Q4_K_M).

Note: This is a GGUF conversion of a model under the NVIDIA Open Model License. Ensure your use complies with that license.

Limitations

The bf16 GGUF (17.79 GB) preserves full precision but is large — needs 24 GB+ VRAM.
The Q4_K_M GGUF (6.53 GB) fits on a single 16 GB GPU but trades some precision for size.
The base model's general capabilities are retained; this is a direct conversion with no additional fine-tuning.