NVIDIA-Nemotron-3-Nano-4B-FP8

Model Overview

License/Terms of Use

Evaluation Results:

Deployment Geography: Global

Use Case

Release Date: 3/16/2026

References

• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
• NVIDIA Nemotron 3: Efficient and Open Intelligence
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

Model Architecture

• Architecture Type: Mamba2-Transformer Hybrid
• Network Architecture: Nemotron-Hybrid
  • This model was compressed from nvidia/NVIDIA-Nemotron-Nano-9B-v2
  • Number of model parameters 3.97 x 10^9

Input

• Input Type(s): Text
• Input Format(s): String
• Input Parameters: One-Dimensional (1D): Sequences
• Other Properties Related to Input: Context length up to 262K. Supported languages include English.

Output

• Output Type(s): Text
• Output Format: String
• Output Parameters: One-Dimensional (1D): Sequences
• Other properties Related to Output: Sequences up to 262K

Software Integration

• Runtime Engine(s): NeMo 25.07
• Supported Hardware Microarchitecture Compatibility: NVIDIA A10G, NVIDIA H100-80GB, NVIDIA A100, GeForce RTX
• Operating System(s): Linux

Use it with Transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8")
model = AutoModelForCausalLM.from_pretrained(
    "nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)

messages = [
    {"role": "system", "content": <system_prompt>},
    {"role": "user", "content": "Write a haiku about GPUs"},
]
tokenized_chat = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    tokenized_chat,
    max_new_tokens=32,
    eos_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(outputs[0]))

messages = [
    {"role": "system", "content": <system_prompt>},
    {"role": "user", "content": "Write a haiku about GPUs"},
]
tokenized_chat = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    enable_thinking=False,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    tokenized_chat,
    max_new_tokens=32,
    eos_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(outputs[0]))

Use it with vLLM

vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron3-nano-4B-FP8\
  --max-num-seqs 8 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --port 8000 \
  --trust-remote-code \
  --mamba_ssm_cache_dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3 \
  --kv-cache-dtype fp8

Use it with TRT-LLM

Model Version

• v1.0

Training, Testing, and Evaluation Datasets

Training datasets

• Data Modality: Text
• Text Training Data Size: More than 10 Trillion Tokens
• Train/Test/Valid Split: We used 100% of the corpus for pre-training and relied on external benchmarks for testing.
• Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
• Labeling Method by dataset: Hybrid: Automated, Human, Synthetic

Public Datasets

Private Non-publicly Accessible Datasets of Third Parties

Online Dataset Sources

NVIDIA-Sourced Synthetic Datasets

Evaluation Dataset:

• Data Collection Method by dataset: Hybrid: Human, Synthetic
• Labeling Method by dataset: Hybrid: Automated, Human, Synthetic

Inference

• Engines: HF, vLLM, llama-cpp, TRT-LLM, SGLang
• Test Hardware: NVIDIA GeForce RTX, H100 80GB, DGX Spark, Jetson Thor/Orin Nano

Ethical Considerations