NVIDIA Cosmos™ is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
Model Overview: Cosmos3-Super-Image2Video-4Step
Description
Cosmos3-Super-Image2Video-4Step is a 4-step distilled version of the base Cosmos3-Super-Image2Video model. Given an input image plus an optional text prompt, it generates a temporally coherent video.
What's different from the base model: Distilled into a fixed 4-step sampling schedule without the need for classifier-free guidance. At the recommended 50-step + CFG setting for the Image2Video model, this replaces 50 × 2 = 100 model evaluations with 4, leading up to 25× fewer diffusion-model evaluations per video (see PBR for latency estimates).
Quality: Ranked #1 among open-source models on the Artificial Analysis Image-to-Video leaderboard (as of 2026/07/17; see Benchmarks section).
Input / Output: Image(s) in → MP4 video out. Recommended 480p (832×480, 16:9); 189 frames by default.
Hardware: The 64B model needs a multi-GPU H100/H200 node (4–8 GPUs) or a single B200; it does not fit on a single smaller GPU. See Usage. Full model details below.
Cosmos3-Super-Image2Video-4Step was developed by NVIDIA as a part of Cosmos3.
Cosmos3 is a collection of Omnimodal world models capable of generating dynamic, high-quality video, image, audio, and action commands from combinations of text, image, video, and action trajectory inputs. It serves as a foundational building block for a broad range of Physical AI applications and research spanning world understanding, world generation, simulation, and embodied policy learning.
This model is ready for commercial and non-commercial use.
Model Developer: NVIDIA
Model Versions
Released on: 07/20/2026
Cosmos3-Edge:
Given multimodal inputs including text, images, video, and action trajectories, generate coherent text, images, video, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
Cosmos3-Edge-Policy-DROID:
Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
Cosmos3-Super-Image2Video-4Step:
Given one or more input images and optional text instructions, generate temporally coherent video sequences that are consistent with the provided visual content. Distilled from Cosmos3-Super-Image2Video using Improved Distribution Matching Distillation (DMD2), enabling high-quality generation in 4 steps.
Cosmos3-Super-Text2Image-4Step:
Given text input, generate high-fidelity images that are consistent with the provided description. Distilled from Cosmos3-Super-Text2Image using Improved Distribution Matching Distillation (DMD2), enabling high-quality generation in 4 steps.
Released on: 05/31/2026
Cosmos3-Nano:
Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
Cosmos3-Super:
Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
Cosmos3-Nano-Policy-DROID:
Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
Cosmos3-Super-Image2Video:
Given one or more input images and optional text instructions, generate temporally coherent video sequences that are consistent with the provided visual content.
Cosmos3-Super-Text2Image:
Given text input, generate high-fidelity images that are consistent with the provided description.
Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising. This unified architecture enables Cosmos3 to model heterogeneous modalities within a single framework while preserving generation mechanisms best suited to each modality.
Input Type(s): Text, Image, Video (with audio or without audio), Action Trajectory
Input Format(s):
Text: String
Image: jpg, png, jpeg, webp
Video (with or without audio): mp4
Action: json (1D list)
Input Parameters:
Text: One-dimensional (1D)
Image: Two-dimensional (2D)
Video: Three-dimensional (3D)
Audio: One-dimensional (1D)
Action trajectory: One-dimensional (1D)
Other Properties Related to Input:
For video inputs, we accept various resolutions, including 720p, 480p, and 256p.
When using input video with audio muxed into the video MP4 file, the audio should have 2 channels (stereo) and a 48 kHz sample rate.
Image and video inputs are RGB color (8 bits per channel, sRGB color space); grayscale inputs are not supported.
Action input is a per-frame sequence of robot/agent state or control values (e.g., joint positions, gripper state, camera pose). The full input is a 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality listed below.
Input action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
Input Size and Length limits:
Text: 4096 tokens
Image: 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
Video: 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16). Max number of frames = 5.
Audio: Max 0.5 second
Action: 16 – 400 video frames
Generator Output
Output Type(s): Image, video, audio, action, text
Output Format(s):
Image: JPG
Video: MP4
Audio: Advanced Audio Coding (AAC) stream (muxed within the MP4)
Action: 1D list (.json)
Text: string
Output Parameters:
Image: Two-dimensional (2D)
Video: Three-dimensional (3D)
Audio: One-dimensional (1D)
Action: One-dimensional (1D)
Text: One-dimensional (1D)
Other Properties Related to Output:
The generated video is an MP4 file, with the resolution, frame rate, and duration specified in the input. The generated audio is encoded in AAC format, muxed into the video MP4 file with 2 channels (stereo) and a 48 kHz sample rate.
Video generation supports durations from 5 to 400 frames, with 189 frames as the default generation duration.
The generated action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
Audio: 48 kHz stereo AAC stream muxed into video mp4
Video: mp4 at the FPS specified in input
Image: JPEG
Reasoner Input
Input Type(s): Text, Text+Image, Text+Video
Input Format(s):
Text: String
Image: jpg, png, jpeg, webp
Video: mp4
Input Parameters:
Text: One-dimensional (1D)
Image: Two-dimensional (2D)
Video: Three-dimensional (3D)
Other Properties Related to Input:
Video inputs are recommended at a frame rate of 4 fps.
Long-context inputs supported up to 256K tokens.
Input Size and Length limits:
Text: Up to 256K tokens (context window).
Image: Standard input image formats; passed as file or URL.
Video: mp4 at the recommended 4 fps.
Reasoner Output
Output Type(s): Text
Output Format(s):
Text: string
Output Parameters:
Text: One-dimensional (1D)
Other Properties Related to Output:
Default max_tokens=4096+ is recommended for reasoning outputs; longer outputs may be requested.
Reasoning outputs may include structured chain-of-thought, 2D/3D point localization, and bounding-box coordinates for vision-based tasks.
The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Linux (We have not tested on other operating systems.)
Note: Only BF16 precision is tested. Other precisions like FP4, FP8, and FP16 are not officially supported.
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Training, Testing, and Evaluation Datasets
Dataset Overview
Total Size: 1.3B data points
Total Number of Datasets: 393 dataset entries
Dataset partition: Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
Time period for training data collection: 2024–2026
Time period for testing data collection: N/A (standard public benchmarks)
Time period for validation data collection: N/A (standard public benchmarks)
Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.
Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.
Data Modality and Training Data Size
Modality
Reasoning Data Sample Count
Generation Data Sample Count
Text
22M
Not Applicable
Image
19M
767M
Video
1M
348M
Audio
Not Applicable
139M
Action
Not Applicable
7M
Data Collection Method by dataset
Hybrid: Automatic/Sensors, Synthetic, Automated
Labeling Method by dataset
Hybrid: Human, Automated
Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
Public Datasets
Dataset
Samples
OpenImage
1.2M
Coyo700M
100M
YouTube Video
340M
UMI
4.5M
Private Datasets
Dataset
Samples
Egocentric
7M
Nexar
0.6M
AgiBot
0.2M
HOI
0.3M
Synthetic Datasets
Dataset
Samples
synthetic images generated using HiDream-I1
15M
synthetic images generated using Qwen-Image-2512
14M
synthetic captions generated using Qwen3-VL
1115M
Evaluation Datasets
Data Collection Method by dataset
Hybrid: Automatic/Sensors, Synthetic, Automated
Labeling Method by dataset
Hybrid: Human, Automated
Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
Benchmarks
Artificial Analysis Leaderboard
Open-Source Models [2026/07/17]
Artificial Analysis Image-to-Video leaderboard (no audio) — open-source models
All Models [2026/07/17] (Including Closed-Source)
Artificial Analysis Image-to-Video leaderboard (no audio) — all models including closed-source
PBR (Performance Benchmark Reporting)
The following tables report estimated inference latency for Cosmos3-Super-Image2Video-4Step, derived from the available Cosmos3-Super Image-to-Video generator benchmarks. The source benchmarks were collected using BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings.
The original Cosmos3-Super Image-to-Video measurements used 35 denoising steps with classifier-free guidance (CFG), requiring two model evaluations per denoising step. Cosmos3-Super-Image2Video-4Step uses a fixed 4-step distilled schedule without CFG. The latency estimates below therefore apply a theoretical 17.5× speedup:
These values are derived estimates rather than direct measurements of the 4-step checkpoint. Actual end-to-end latency may differ because input-image processing, text encoding, VAE encoding and decoding, output encoding, GPU communication, and other fixed overheads do not necessarily scale with the number of denoising steps.
Latency is reported in seconds. The suffix in each resolution column indicates the number of GPUs used; for example, 480p/4 represents 480p generation using four GPUs. Cells marked N/A indicate configurations unavailable because the model does not fit in GPU memory, while Not tested indicates configurations for which the source benchmark did not report a measurement.
PyTorch
GPU
480p/1
480p/4
480p/8
RTX PRO 6000 Blackwell
N/A
11.558
6.797
H20
N/A
15.899
8.754
H100 NVL
N/A
5.689
3.667
H200 NVL
N/A
4.715
2.720
H200 141GB HBM3
N/A
4.026
2.406
B200
6.423
2.040
1.214
vLLM-Omni
GPU
480p/1
480p/4
480p/8
RTX PRO 6000 Blackwell
N/A
Not tested
Not tested
H20
N/A
Not tested
Not tested
H100 NVL
N/A
Not tested
Not tested
H200 NVL
14.531
4.103
2.201
H200 141GB HBM3
12.611
3.731
2.109
B200
6.581
2.114
1.329
Benchmark notes:
All estimates use the same source workload, including the same prompt, seed, and sampler settings.
Results use BF16 precision and batch size 1.
Configurations labeled /4 and /8 use four and eight GPUs, respectively. PyTorch uses automatic sharding, while vLLM-Omni uses HSDP/Ulysses for multi-GPU execution.
PyTorch source results report average generation or sampling time from OSS inference benchmarking.
vLLM-Omni source results report "Avg. Generation Time" for the corresponding vLLM-Omni inference configuration.
Cells marked N/A indicate configurations unavailable because the 64B model does not fit on a single GPU. Cells marked Not tested indicate configurations for which no source measurement was reported.
All displayed values were calculated as Cosmos3-Super source latency ÷ 17.5 and rounded to three decimal places.
Note on speedup basis: the 17.5× above is extrapolated from a 35-step base, which was the step count used in the original Cosmos3-Super base model speed benchmark. For Image-to-Video specifically, the recommended sampler setting is 50 steps, corresponding to 50 × 2 ÷ 4 = 25× fewer model evaluations; this is the figure used in general reporting. The 17.5× estimate is retained here as the more conservative, benchmark-matched number.
Usage: Prepare Prompts with Prompt Upsampling
Cosmos3-Super-Image2Video-4Step supports dense natural-language prompts together with a first-frame image. Prompt upsampling is optional for distilled models: you can run inference directly with a natural-language prompt, or use a JSON-format upsampled prompt for potential quality improvements. Description and code can be found here.
The JSON-upsampled version of assets/original_prompt.txt is saved in assets/example_prompt.json for convenience, and is used for the video generation examples below.
You can use the release-tested vllm-omni package for deploying an OpenAI-compatible API inference endpoint.
The following configuration uses 4 GPUs and shards the 64B model with HSDP. This configuration is suitable for H100/H200-class multi-GPU nodes when sufficient memory is available:
For an 8-GPU node, use --ulysses-degree 4 --hsdp-shard-size 8. Tensor parallelism is also supported by setting --tensor-parallel-size. Setting --enable-layerwise-offload can help reduce memory usage on GPUs with less available memory.
Single-GPU Serving
On a single GPU with sufficient memory, such as B200, serve the model without distributed parallelism or HSDP flags:
Single-GPU serving requires enough device memory for the 64B model. GPUs with less memory, such as H100/H200-class devices, may run out of memory; use the multi-GPU configuration above for those cases.
Cosmos3-Super-Image2Video-4Step uses a fixed 4-step distilled schedule. vLLM-Omni detects the distilled checkpoint from scheduler/scheduler_config.json and reads the schedule from fixed_step_sampler_config.t_list. num_inference_steps and flow_shift are not user-configurable request fields for this model. Use guidance_scale=1.0.
Download Example Prompts and Scripts
The inference scripts (scripts/) and example inputs (assets/) live in this model repo. Download just those folders with the Hugging Face CLI:
Run all commands below from the downloaded repo root.
Example: Image to Video Generation
Generate a video from a first-frame image and a JSON-format prompt by calling the vLLM-Omni endpoint. The examples below use 480p generation (832x480 at 16:9), the recommended setting for this distilled I2V checkpoint. By default, the script uses http://localhost:8000, which matches the serving commands above. If the server runs on another host, pass --endpoint http://<server-host>:8000.
1import json
2import requests
34# 1. Read JSON-upsampled prompt.5json_prompt = json.load(open("assets/example_prompt.json"))67# 2. Build the request body.8request_body ={9"prompt": json_prompt["prompt"],10"negative_prompt": json_prompt["negative_prompt"],11"size":"832x480",12"num_frames":189,13"fps":24,14"guidance_scale":1.0,15"extra_params": json.dumps(16{17"use_resolution_template":False,18"use_duration_template":False,19"guardrails":True,20}21),22}23files ={"input_reference":("input.png",open("assets/example_first_frame.png","rb"),"image/png")}2425# 3. Send the POST request.26url ="http://localhost:8000/v1/videos/sync"27print("Sending request to server...")28response = requests.post(url, data=request_body, files=files, headers={"Accept":"video/mp4"})29response.raise_for_status()3031# 4. Save the returned MP4 bytes.32withopen("/tmp/cosmos3_i2v_4step.mp4","wb")as video_file:33 video_file.write(response.content)34print("Saved video to /tmp/cosmos3_i2v_4step.mp4")
Example output generated from assets/example_first_frame.png:
Usage: Run Inference with Diffusers
Cosmos3-Super-Image2Video-4Step is supported by the Hugging Face Diffusers modular pipeline. The distilled checkpoint is not compatible with Cosmos3OmniPipeline or Cosmos3OmniModularPipeline; it must be loaded with Cosmos3DistilledModularPipeline.
Install
Install Diffusers from the main branch until a release containing Cosmos3DistilledModularPipeline is available:
The 4-step schedule and the guidance scale come from the checkpoint: num_inference_steps is fixed to the length of the distilled sigma schedule and guidance_scale is forced to 1.0, so passing any other value raises an error. negative_prompt is ignored, and unlike the base Cosmos3 models there is no scheduler to swap in and no flow_shift to set.
Example output generated from assets/example_first_frame.png:
Inferencing with Custom Prompts
Cosmos3-Super-Image2Video-4Step supports dense natural-language prompts together with a first-frame image, and it also supports JSON-format upsampled prompts for improved quality. The recommended way to prepare an upsampled prompt is to use Prepare Prompts with Prompt Upsampling. The helper below is a simple proof-of-concept script for building a JSON-format prompt from a custom natural-language instruction and the input image. It requires an OpenAI-compatible VLM model like claude-opus-4.7 and gpt-5.5.
bash
1exportPROMPT_UPSAMPLER_API_KEY="..."2python scripts/upsample_prompt.py \3 --model-name <model>\4 --base-url <VLM-endpoint-url>\5 --image-path assets/example_first_frame.png \6 --user-prompt "The ice cream melts and gradually disappears. The camera moves around."\7 --output-path scripts/upsampled.json
Limitations
Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
Test Hardware: RTX PRO 6000, H20, H100, H200, B200
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
