At a Glance
| |
|---|
| Total parameters | 31B (Mamba2-Transformer hybrid MoE) |
| Active parameters | ~3B per token |
| Modalities (in) | Video, Audio, Image, Text; video with its soundtrack in one pass |
| Modality (out) | Text |
| Reasoning mode | On by default |
| Quantization | Q4_K_M (GGUF); projector F16 (audio tower F16/BF16) |
| Files | Nine quantizations (Q3_K_M, IQ3_M, IQ4_XS, MXFP4_MOE, Q4_K_M, Q5_K_M, Q6_K, Q8_0, BF16) plus the unified projector mmproj-omni-vision-audio-F16.gguf (2.8 GB); single-tower projectors kept for compatibility. Every quantization is verified on image, audio, and video with soundtrack: see TEST-MATRIX.md |
| Runtime | llama.cpp: audio and video require the llama.cpp-omni fork (feat/nemotron-video); image and text run on stock |
| Minimum hardware | Any system llama.cpp supports with ~28 GB free memory; verified on NVIDIA DGX Spark (GB10, unified memory) |
Model Parameters
| Mode | temperature | top_p | max_tokens |
|---|
| Thinking mode | 0.6 | 0.95 | 20480 |
| Transcription / deterministic | 0.0 | : | 2048 |
Model Overview
Description:
NVIDIA Nemotron 3 Nano Omni is a multimodal large language model that unifies video, audio, image, and text understanding to support enterprise-grade Q&A, summarization, transcription, and document intelligence workflows. It extends the Nemotron Nano family with integrated video+speech comprehension, Graphical User Interface (GUI), Optical Character Recognition (OCR), and speech transcription capabilities. NVIDIA Nemotron 3 Nano Omni was developed by NVIDIA as part of the Nemotron model family.
This repository provides a GGUF release of the model for llama.cpp, quantized to Q4_K_M, together with a unified multimodal projector: a single mmproj GGUF carrying both the C-RADIO v4-H vision encoder (image and video frames) and the Parakeet speech encoder (audio inputs). Both towers are packaged in the mixed-modality GGUF layout llama.cpp resolves per modality (clip.vision.projector_type = nemotron_v2_vl, clip.audio.projector_type = parakeet), so one server process accepts video, audio, image, and text without swapping projector files. A video that carries an audio track is consumed in one pass: the frames go to the vision tower and the soundtrack is demuxed and fed to the audio tower in the same request, so the model answers about what it sees and what it hears from a single file.
This model is available for commercial use.
License/Terms of Use
Governing Terms: Use of this model is governed by the
NVIDIA Open Model Agreement Deployment Geography:
Global
Use Case:
This release is intended for users who require the Nemotron 3 Nano Omni multimodal capabilities: video and speech analysis, document intelligence (OCR, charts, long documents), speech transcription: on hardware served by llama.cpp, including single-GPU and unified-memory systems where the PyTorch serving stacks are not practical.
Release Date:
Hugging Face 08/22/2026
Model Architecture:
Architecture Type: Mamba2-Transformer Hybrid Mixture of Experts (MoE)
Network Architecture:
- Nemotron 3 Nano LLM (30B A3B): 31B-parameter Mamba2-Transformer hybrid MoE backbone with ~3B active parameters per token, quantized here to Q4_K_M.
- CRADIO v4-H: vision encoder for image and video frames, F16, in the unified projector.
- Parakeet: speech encoder for audio inputs, F16/BF16, in the unified projector.
Changes from the source checkpoint:
- GGUF conversion of the language model from Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, quantized to Q4_K_M.
- Both encoder towers converted from the same checkpoint and merged into one mixed-modality projector file; encoder weights preserved bit-exact from the per-tower conversions.
- The audio (Parakeet FastConformer) compute graph and the video temporal-patch path are additions to llama.cpp carried in the llama.cpp-omni fork, validated against a PyTorch reference (video graph to 0.0019%, audio to 0.03%). An upstream contribution is planned.
Input(s):
Input Type(s): Video, Audio, Image, Text
Input Format(s):
- Video: mp4
- Audio: wav, mp3 (16 kHz mono recommended for transcription)
- Image: Red, Green, Blue (RGB) (jpeg, png)
- Text: String
Output(s)
Output Type(s): Text
Other Properties Related to Output:
- Supports reasoning output with chain-of-thought (on by default)
Software Integration:
Runtime Engine(s):
- llama.cpp (image, text: stock; audio, video: llama.cpp-omni, branch
feat/nemotron-video)
Supported Hardware Microarchitecture Compatibility:
- Any platform llama.cpp supports (CUDA, Metal, CPU); verified on NVIDIA Blackwell (DGX Spark / GB10)
Preferred/Supported Operating System(s):
Quick Start
1git clone https://github.com/VincentKaufmann/llama.cpp-omni
2cd llama.cpp-omni && git checkout feat/nemotron-video
3cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-mtmd-cli llama-server
1# image
2./build/bin/llama-mtmd-cli -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-Q4_K_M.gguf \
3 --mmproj mmproj-omni-vision-audio-F16.gguf --image photo.jpg -p "Describe this image."
4
5# audio
6./build/bin/llama-mtmd-cli -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-Q4_K_M.gguf \
7 --mmproj mmproj-omni-vision-audio-F16.gguf --audio speech.wav -p "Transcribe this audio."
8
9# video: a soundtrack, when present, is heard in the same pass
10./build/bin/llama-mtmd-cli -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-Q4_K_M.gguf \
11 --mmproj mmproj-omni-vision-audio-F16.gguf --video clip.mp4 \
12 -p "What is shown, and what does the narrator say?"
13
14# server: one process, all modalities on /v1/chat/completions
15./build/bin/llama-server -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-Q4_K_M.gguf \
16 --mmproj mmproj-omni-vision-audio-F16.gguf
Roadmap
The next release rebases this work onto current llama.cpp master and adopts
the upstream audio projector layout. After that, image, text and audio run on
stock llama.cpp with these files; the fork remains required only for video and
one-pass audio+video.
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility. Please refer to the source model's
model card for the training data, evaluation, bias, and safety documentation that applies to these weights. This repository changes the storage format and serving runtime only; it does not alter model behavior beyond the effects of Q4_K_M quantization.
