Based on NVIDIA Nemotron Nano v2 12B VL, converted to GGUF as BF16 using GGML's convert_hf_to_gguf.py and then quantized to 4-bit using llama-quantize, allowing for inference on smaller consumer GPUs via Llama.cpp.

Use Case:

Nemotron Nano 12B V2 VL is a model for multi-modal document intelligence. It would be used by individuals or businesses that need to process documents such as invoices, receipts, and manuals. The model is capable of handling multiple images of documents, up to four images at a resolution of 1k x 2k each, along with a long text prompt. The expected use is for tasks like summarization and Visual Question Answering (VQA). The model is also expected to have a significant advantage in throughput.

Inference with Llama.cpp

Make sure to use the latest release of Llama.cpp.

Using llama-mtmd-cli

BF16

llama-mtmd-cli -m NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.gguf --mmproj mmproj-BF16.gguf -p "Describe this image" --image image.jpg -c 4096 --jinja

Q4_0

llama-mtmd-cli -m NVIDIA-Nemotron-Nano-12B-v2-VL-Q4_0.gguf --mmproj mmproj-BF16.gguf -p "Describe this image" --image image.jpg -c 4096 --jinja

Using llama-server

BF16

llama-server --host 0.0.0.0 --port 8080 -hf tomlawrence/NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF:BF16 --jinja -ngl 99 --ctx-size 16384

Q4_0

llama-server --host 0.0.0.0 --port 8080 -hf tomlawrence/NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF:Q4_0 --jinja -ngl 99 --ctx-size 16384

Access using the Llama.cpp Web UI on the IP address and specified port (e.g. 8080)