Model Architecture: NemotronHForCausalLM
- Input: Text
- Output: Text
- Total Parameters: 550B
- Active Parameters: 55B
Model Optimizations:
- Activation quantization: FP8
- Weight quantization: FP8
Intended Use Cases:
- Reasoning and complex problem solving.
- Mathematics and science.
- Code generation.
- Instruction following.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
Release Date: 06/04/2025
Version: 1.0
Model Developers: Red Hat

Quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.

Model Optimizations

This model was obtained by quantizing the weights and activations of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only the weights and activations of the linear operators within transformer blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend.

Install dependencies:

bash

uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install llmcompressor

Launch the vLLM server:

bash

vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
  --host 0.0.0.0 --port 8088 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 32768 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser nemotron_v3 \
  --mamba-ssm-cache-dtype float16 \
  --mamba-backend flashinfer \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
  --trust-remote-code

Send requests:

python

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8088/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic"

messages = [
    {"role": "user", "content": "Solve for x: 2x + 5 = 13"},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was quantized using the llm-compressor library as shown below.

python

from llmcompressor import model_free_ptq

MODEL_ID = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"

model_free_ptq(
    model_stub=MODEL_ID,
    save_directory=SAVE_DIR,
    scheme="FP8_DYNAMIC",
    ignore=[
        "re:.*gate$",
        "lm_head",
        "model.embed_tokens",
        "re:.*mixer.conv1d.*",
        "re:.*norm_f*",
        "re:.*bias$",
        "re:.*embed_tokens$",
        "backbone.embeddings"
    ],
    max_workers=15,
    device="cuda:0",
)

Evaluation

The model was evaluated on reasoning tasks using lighteval. vLLM was used as the serving backend for all evaluations.

Install dependencies:

bash

uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install lighteval==0.13.0
uv pip install "litellm[caching]>=1.66.0"

Launch the vLLM server:

bash

vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
  --host 0.0.0.0 --port 8088 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 32768 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser nemotron_v3 \
  --mamba-ssm-cache-dtype float16 \
  --mamba-backend flashinfer \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
  --trust-remote-code

AIME 2025:

bash

lighteval endpoint litellm \
  "model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
  "aime25|0" \
  --output-dir results --save-details

GPQA Diamond:

bash

lighteval endpoint litellm \
  "model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
  "gpqa:diamond|0" \
  --output-dir results --save-details

Accuracy

Benchmark	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic (this model)	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16
AIME 2025 (pass@1)	90.00	90.00 (100.0%)	93.33 (103.7%)	86.67 (96.3%)	86.67 (96.3%)
GPQA Diamond (pass@1)	78.79	84.85 (107.7%)	82.32 (104.5%)	81.31 (103.2%)	81.82 (103.8%)
Average	84.39	87.42 (103.6%)	87.83 (104.1%)	83.99 (99.5%)	84.24 (99.8%)

Multi-category Evaluation

Category	Benchmark	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic (this model)	Recovery
Instruct	MMLU-CoT (5-shot)	91.09	91.14	100.05%
Instruct	GSM8K Platinum (5-shot)	98.54	98.70	100.16%
Instruct	IFEval (0-shot)	91.00	90.70	99.67%
Instruct	MATH-500	83.60	83.27	99.61%
Reasoning	AIME 2025	64.17	63.75	99.35%
Reasoning	MATH-500	85.80	86.07	100.31%
Reasoning	GSM8K Platinum (0-shot)	96.77	96.99	100.23%
Reasoning	IFEval (0-shot)	94.58	94.45	99.86%
Coding	LCB CodeGen v6	51.81	50.00	96.51%

Tool Calling Evaluation

The model was evaluated on tool calling tasks using the Berkeley Function-Calling Leaderboard v4 (BFCLv4). vLLM was used as the serving backend for all evaluations.

Category	Benchmark	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16	RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic (this model)	Recovery
Overall	BFCLv4 Overall Acc	55.44	55.53	100.16%
Single Turn	Non-Live Acc	45.00	44.56	99.02%
Single Turn	Live Acc	71.65	72.02	100.52%
Multi-Turn	Multi-Turn Acc	42.12	41.88	99.43%
Agentic	Agentic Acc	57.64	58.14	100.87%

RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic

NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic Model Icon

Model Overview

Model Optimizations

Deployment

Use with vLLM

Creation

Evaluation

Accuracy

Multi-category Evaluation

Tool Calling Evaluation

RedHatAI/
NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic

NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic
Model Icon