Read our How to Run Nemotron 3 Nano Omni Guide!

---

Model Overview

Description:

License/Terms of Use

Deployment Geography:

Use Case:

• Customer service applications (e.g., Doordash video of drop-off at a given address via OCR, drive-thru order verification)
• Media and Entertainment (M&E) — video and speech analysis, dense captions, video search and summarization
• Document intelligence for AI assistants (contracts, SOW/MSA, scientific discovery, financial documents)
• GUI automation for AI agentic applications (incident management, agentic search, browser agents, email agents)

Release Date:

• BF16
  
• FP8
  
• NVFP4
  

Model Architecture:

• Nemotron 3 Nano LLM (30B A3B)
• CRADIO v4-H vision encoder
• Parakeet speech encoder

Input(s):

• Video: mp4, up to 2 minutes. For 1080p videos, sample up to 1 FPS / 128 frames. For lower-resolution videos such as 720p, higher temporal sampling such as 2 FPS / 256 frames may be used.
• Audio: wav, mp3 files (up to 1 hour), 8kHz and higher sampling rates
• Image: Red, Green, Blue (RGB) (jpeg, png)
• Text: String
  

• Video: Three-Dimensional (3D)
• Audio: One-Dimensional (1D)
• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)
  

• Maximum context length up to 256k tokens
• Language support: English only
  

Output(s)

• Text: String
  

• Text: One-Dimensional (1D)
  

• Maximum context length up to 256k tokens.
• Supports JSON output format
• Supports reasoning output with chain-of-thought
• Supports tool calling
• Supports word-level timestamps for transcription
  

Software Integration:

• vLLM
  
• NeMo
  
• Megatron
  
• NeMo-RL
  

• NVIDIA Ampere (A100 80GB SXM/NVLink)
  
• NVIDIA Blackwell (B200 SXM/NVLink, RTX Pro 6000 SE, DGX Spark, Jetson Thor, RTX 5090)
  
• NVIDIA Hopper (H100 SXM/NVLink, H200 SXM/NVLink)
  
• NVIDIA Lovelace (L40S)
  

• Linux
  

• vLLM
  
• TensorRT LLM
  
• TensorRT Edge-LLM
  
• llama.cpp
  
• Ollama
  
• SGLang
  

Model Version(s):

---

Quick Start Guide

Model Parameters

Download Model Weights

Install the HuggingFace CLI

Download the weights

• hf download is resumable — re-run the same command if the connection drops.
• --max-workers 8 parallelizes downloads; tune up on fast networks.
• The hf_xet extra enables native Xet-protocol transfers for Xet-backed repos; no need for git-xet or git-lfs when using hf download.

Verify the download

---

vLLM

Required version: vLLM 0.20.0 is needed. This means one of these containers:

• CUDA 13.0: 'vllm/vllm-openai:v0.20.0'
• CUDA 12.9: 'vllm/vllm-openai:v0.20.0-cu129'

Container

Audio support: Within the vLLM container, before running vllm serve, if any audio will be used (including passing use_audio_in_video: true):

python3 -m pip install "vllm[audio]"

General Invocation (1×GPU, e.g. 1×B200)

Platform-Specific Notes

RTX Pro: Due to a current bug with FlashInfer + RTX Pro, append: --moe-backend triton

NVFP4 + TP>1: Due to a current bug with the TRTLLM_GEN MoE backend kernels on vLLM, when running with TP>1 on NVFP4, append: --moe-backend flashinfer_cutlass

vLLM on DGX Spark (aarch64 / ARM64)

1. Pull the container image

2. Launch the vLLM server on Spark

Key Spark-Specific Flags

Memory Tuning on Spark

1. Lower --gpu-memory-utilization from 0.85 → 0.70 to free ~19 GB back to the OS and re-enable weight prefetch. Cost: smaller KV cache budget.
2. Lower --max-model-len to reduce KV cache allocation (e.g. halving context window halves KV cache at --max-num-seqs=1). Combined override:

---

TensorRT-LLM

Platform-Specific Notes

TensorRT Edge-LLM

---

SGLang

• CUDA 13.0: lmsysorg/sglang:dev-cu13-nemotronh-nano-omni-reasoning-v3
• CUDA 12.9: lmsysorg/sglang:dev-nemotronh-nano-omni-reasoning-v3

[!NOTE] NVFP4 and FP8 support to come.

Platform-Specific Notes

SGLang on DGX Spark (aarch64 / ARM64)

1. Pull the container image

2. Launch the SGLang server on Spark

Key Spark-Specific Flags

Memory Tuning on Spark

1. Lower --mem-fraction-static from 0.80 → 0.70 to free ~13 GB back to the OS and re-enable weight prefetch. Cost: smaller KV cache budget.
2. Lower --context-length to reduce KV cache allocation (e.g. halving context window halves KV cache at --max-running-requests=1). Combined override:

---

API Client (OpenAI-compatible)

pdf_vlm_chat.py (click to expand)

python

1#!/usr/bin/env python3 2"""Send PDF page(s) as images to a vLLM /v1/chat/completions endpoint.""" 3from __future__ import annotations 4 5import argparse, base64, sys 6from io import BytesIO 7from pathlib import Path 8 9import requests 10 11try: 12 import fitz 13 from PIL import Image 14except ImportError: 15 print("Install: pip install pymupdf pillow requests", file=sys.stderr) 16 sys.exit(1) 17 18USER_PROMPT = ( 19 "Summarize this PDF page: main topic, section headings, important facts " 20 "or bullets, and a brief note on each figure or table. " 21 "Do not invent text you cannot read." 22) 23API_URL = "http://localhost:8000/v1/chat/completions" 24MODEL = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4" 25MAX_TOKENS = 32000 26DPI = 150 27 28 29def page_to_b64(pdf_path: str, idx: int) -> str: 30 doc = fitz.open(pdf_path) 31 z = DPI / 72.0 32 pix = doc.load_page(idx).get_pixmap(matrix=fitz.Matrix(z, z)) 33 img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples) 34 doc.close() 35 buf = BytesIO() 36 img.save(buf, format="PNG") 37 return base64.b64encode(buf.getvalue()).decode("ascii") 38 39 40def chat(url, model, b64, text, max_tokens): 41 r = requests.post(url, json={ 42 "model": model, 43 "messages": [{"role": "user", "content": [ 44 {"type": "text", "text": text}, 45 {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}, 46 ]}], 47 "max_tokens": max_tokens, 48 "stream": False, 49 "temperature": 1.0, 50 "chat_template_kwargs": {"enable_thinking": False}, 51 }, timeout=120) 52 r.raise_for_status() 53 return r.json()["choices"][0]["message"]["content"] 54 55 56def main(): 57 p = argparse.ArgumentParser() 58 p.add_argument("pdf") 59 p.add_argument("--page", type=int, default=0) 60 p.add_argument("--all-pages", action="store_true") 61 p.add_argument("-o", "--output") 62 p.add_argument("--url", default=API_URL) 63 p.add_argument("--model", default=MODEL) 64 p.add_argument("--max-tokens", type=int, default=MAX_TOKENS) 65 a = p.parse_args() 66 67 doc = fitz.open(a.pdf); n = len(doc); doc.close() 68 pages = range(n) if a.all_pages else [a.page] 69 parts = [f"# Extracted: {Path(a.pdf).name}\n\n*Pages: {n}*\n"] if a.all_pages else [] 70 71 for i in pages: 72 print(f"Page {i+1}/{n} ...", file=sys.stderr) 73 b64 = page_to_b64(a.pdf, i) 74 text = chat(a.url, a.model, b64, f"Page {i+1}.\n\n{USER_PROMPT}", a.max_tokens) 75 parts.append(f"\n---\n\n## Page {i+1}\n\n{text.strip()}\n" if a.all_pages else text.strip()) 76 77 out = "\n".join(parts) 78 if a.output: 79 Path(a.output).write_text(out + "\n", encoding="utf-8") 80 else: 81 print(out) 82 83if __name__ == "__main__": 84 main()

---

Reasoning Mode (enable_thinking)

---

Advanced: Budget-Controlled Reasoning

python

1from typing import Any, Dict, List 2 3from openai import OpenAI 4from transformers import AutoTokenizer 5 6 7class ThinkingBudgetClient: 8 def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str): 9 self.tokenizer = AutoTokenizer.from_pretrained( 10 tokenizer_name_or_path, trust_remote_code=True 11 ) 12 self.client = OpenAI(base_url=base_url, api_key=api_key) 13 14 def chat_completion( 15 self, 16 model: str, 17 messages: List[Dict[str, Any]], 18 reasoning_budget: int = 512, 19 max_tokens: int = 1024, 20 **kwargs, 21 ) -> Dict[str, Any]: 22 assert max_tokens > reasoning_budget, ( 23 f"reasoning_budget must be less than max_tokens. " 24 f"Got {max_tokens=} and {reasoning_budget=}" 25 ) 26 27 # Step 1: generate only the reasoning trace up to the requested budget. 28 response = self.client.chat.completions.create( 29 model=model, 30 messages=messages, 31 max_tokens=reasoning_budget, 32 extra_body={ 33 "top_k": 1, 34 "chat_template_kwargs": { 35 "enable_thinking": True, 36 }, 37 }, 38 **kwargs, 39 ) 40 reasoning_content = response.choices[0].message.content or "" 41 if "</think>" not in reasoning_content: 42 print("No </think> found in reasoning content") 43 reasoning_content = f"{reasoning_content}</think>\n\n" 44 45 reasoning_tokens_len = len( 46 self.tokenizer.encode(reasoning_content, add_special_tokens=False) 47 ) 48 remaining_tokens = max_tokens - reasoning_tokens_len 49 assert remaining_tokens > 0, ( 50 f"No tokens remaining for response ({remaining_tokens=}). " 51 "Increase max_tokens or lower reasoning_budget." 52 ) 53 54 # Step 2: continue from the closed reasoning trace and ask for the final answer. 55 continued_messages = messages + [ 56 {"role": "assistant", "content": reasoning_content} 57 ] 58 prompt = self.tokenizer.apply_chat_template( 59 continued_messages, 60 tokenize=False, 61 continue_final_message=True, 62 ) 63 response = self.client.completions.create( 64 model=model, 65 prompt=prompt, 66 max_tokens=remaining_tokens, 67 extra_body={"top_k": 1}, 68 **kwargs, 69 ) 70 71 return { 72 "reasoning_content": reasoning_content.strip(), 73 "content": response.choices[0].text, 74 "finish_reason": response.choices[0].finish_reason, 75 }

Video Tuning

Frame sampling (--media-io-kwargs)

---

Notes

1. Reasoning default: Reasoning is on by default. If you omit chat_template_kwargs, the model will produce chain-of-thought traces in content. This is appropriate for text and image inputs.
2. Video frame sampling: The default (~32 frames) is too conservative for most real videos. Set --media-io-kwargs at server launch.
3. PDF input format: The API does not accept raw PDF uploads. Render pages to PNG and send as base64 (see PDF Example above).
4. max_tokens vs --max-model-len: max_tokens in the request caps only the completion (generated output). It cannot exceed the server's --max-model-len, which is the hard ceiling for prompt + completion combined. Increase the server flag if you need longer outputs.

---

Jetson Deployment

---

Training, Testing, and Evaluation Datasets:

Dataset Overview

Dataset Description

Public Datasets

Private Datasets

Self-Sourced Synthetic Data

• Overall Size: 41,502,625 samples across modalities: text+audio, text+image, text+video
  
• Description of synthetic data generation methods:
  

NVIDIA-Sourced Synthetic Datasets

Training Dataset:

• Audio
  
• Image
  
• Text
  
• Video
  

• 10,000 to 1 Million Hours
  (267,898,865 audio-containing samples)
  

• 1 Million to 1 Billion Images
  (70,143,901 image-containing samples)
  

• 1 Billion to 10 Trillion Tokens
  (~717.0B tokens total across all modalities)
  

• 10,000 to 1 Million Hours
  (24,557,717 video-containing samples)
  

• Hybrid: Human, Automated, Synthetic
  

• Hybrid: Human, Automated, Synthetic
  

Evaluation Dataset:

• Hybrid: Human, Automated — Evaluation benchmarks are primarily human-curated public academic datasets with automated scoring.
  

• Human
  

Inference:

• NVIDIA H100 SXM
  
• NVIDIA H200 SXM
  
• NVIDIA B200 SXM
  
• NVIDIA A100 80GB SXM
  
• NVIDIA GB200 NVL72
  
• NVIDIA RTX PRO 6000 SE Blackwell
  
• NVIDIA L40S PCIe 48GB
  
• NVIDIA DGX Spark
  
• NVIDIA Jetson Thor
  
• NVIDIA RTX 5090

Best Practices

Sampling Parameters

• Thinking mode for long document analysis and multimodal reasoning tasks:
  temperature=0.5-0.7, top_p=0.95, grace_period=1024, reasoning_budget=16384, max_token=20480, and max_model_len=210000
  
• Instruct mode (non-thinking) for general tasks:
  temperature=0.2, top_k=1
  
• For ASR tasks, we recommend non-thinking mode with temperature=0.2, top_k=1
  

Model output length

Ethical Considerations: