NL-Diffusion-Image introduces a new paradigm for high-resolution text-to-image generation via LLM based on masked discrete diffusion over tokenized image patches.
Each image is encoded into a sequence of discrete tokens (using a 128K codebook/vocabulary), and generation proceeds through iterative parallel unmasking - similar to Diffusion LLMs.
We finetune from Nemotron-Labs-Diffusion and introduce 2 key components:
A token-editing mechanism that allows the model to revise already-unmasked tokens during inference.
Grouped Cross-Entropy (GCE) objective to handle large-vocabulary training efficiently.
NL-Diffusion-Image Paper: Shufan Li et al., "Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis,".
Nemotron-Labs-Diffusion Paper: Yonggan Fu et al., "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding".
Emu3.5 Paper: Emu3.5 team, "Emu3.5: Native Multimodal Models are World Learners".
Model Architecture
Architecture Type: Neural Network Network Architecture: Masked Diffusion Transformer, IBQ tokenizer for visual encoding/decoding Number of model parameters: ~8B parameters
We encode 16x16 image patches using a pretrained discrete tokenizer from Emu3.5, with a codebook size of 128k token IDs.
We expand the Nemotron-Labs-Diffusion vocabulary with a corresponding number of randomly-initialized embeddings, and fine-tune the model on a dataset of image/caption pairs.
Computational Load (Internal Only: For NVIDIA Models Only)
Estimated using Compute ≈ 6ND, with N the number of trained parameters (encoder + training-only decoder ≈ 1.45B) and D the number of patch tokens processed during training.
With 58B tokens for pre-training, 255B tokens for Supervised Fine Tuning, D ≈ 3.1×10¹¹ tokens (image and text included).
Cumulative Compute: ≈ 1.5e22 FLOPS. Estimated Energy: ≈ 53000 kWh. Estimated Emissions for Model Training: ≈ 22.4 tCO2e.
Input
Input Type(s): Text Input Format(s): Character Strings Input Parameters: One-dimensional (1D) Other Properties Related to Input: Maximum prompt length is 900 tokens.
Output
Output Type(s): Images Output Format: Tensor (3xHxW) Output Parameters: Two-Dimensional (2D) Other Properties Related to Output: The output represents an RGB image.
Software Integration
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment.
Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.
A large, filtered collection of diverse real-world web images - including people, animals, objects, scenes, artwork, products, documents, and memes - paired with high-quality synthetic text captions generated from the images.
Data Modality:
Multimodal (image, text)
Image Training Data Size:
115M samples
Data Collection Method by dataset:
Automated
Labeling Method by dataset:
Synthetic: Subset of 8M images recaptioned using Qwen3-VL
MidJourney v6 520k Recaptioned
High-quality synthetic Midjourney v6 images from user prompts.
Data Modality:
Multimodal (image, text)
Image Training Data Size:
520k samples
Data Collection Method by dataset:
Automated
Labeling Method by dataset:
Synthetic: Images recaptioned using Qwen3-VL
COYO700M Recaptioned
Image–alt-text pairs from HTML pages.
Data Modality:
Multimodal (image, text)
Image Training Data Size:
700M samples
Data Collection Method by dataset:
Automated
Labeling Method by dataset:
Synthetic: Subset of 24M images recaptioned using Qwen3-VL
BLIP3o-60k Recaptioned
Data Modality:
Multimodal (image, text)
Image Training Data Size:
520k samples
Data Collection Method by dataset:
Automated
Labeling Method by dataset:
Synthetic: Images recaptioned using Qwen3-VL
Evaluation Datasets
ImageNet
Labeled natural images of objects and scenes organized by WordNet categories.
Text-to-Image Generation Performance on DPG Benchmark and MJHQ-30k Dataset
Model
Params
Codebook
DPG
MJHQ FID
MJHQ HPSv3
MMaDa
8B
8,192
53.4
32.85
5.43
LaViDa-O
10B
8,192
81.8
6.68
8.81
Nemotron-Labs-Diffusion-Image
8B
131,072
85.2
6.46
9.61
Nemotron-Labs-Diffusion-Image*
8B
131,072
86.9
12.23
10.76
* Finetuned on 6M synthetic data for better image quality
Key Considerations
This model can generate synthetic images and may produce content that is inaccurate, offensive, or otherwise inappropriate. Users should implement robust safety guardrails — including content filtering, abuse monitoring, and access controls— to reduce the risk of harmful outputs. Users are responsible for ensuring that their use of the model complies with all applicable laws and regulations, and for regularly reviewing and updating their guardrails as risks evolve. For more information about the implementation of guardrails, please see the Nemotron-3.5-Content-Safety Guardrail Model.
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
Quick Start
import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
model_path = "shufanlNvidia/NLD-Diffusion-Image-8B-Internal"
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
)
model = model.to("cuda").eval()
model.config.dlm_paradigm = "bidirectional"
PROMPT = (
"A full-body shot of hyper-realistic female cyborg, human facial skin seamlessly integrated with a glossy white mechanical head shell. "
"Features a realistic human ear, blue eyes. bright, outdoor, background with blue sky, illuminated by striking bright white studio lighting, "
"casting soft shadows. Cyberpunk aesthetic, high-tech minimalism, shot on 85mm lens, photorealistic, Unreal Engine 5 render, intricately detailed, "
"8k resolution, high dynamic range, chest with whit armor plate, cute, beautiful, sexy, glossy surface, reflective, Artstation, pixiv, no hair, "
"3D render, stylized eyesz"
)
torch.manual_seed(42)
image = model.text_to_image(
PROMPT,
tokenizer=tokenizer,
image_resolution=1024,
n_tokens=(1024 // 16) * (1024 // 16),
guidance_scale=5.0,
temperature=0.86,
n_steps=64,
schedule="shift",
shift=5,
confidence_policy="mmada",
schedule_temp="linear",
alg_temp=1.0,
dynamic_temperature=False,
min_temperature=0.01,
edit_threshold=0.6,
micro_cond="ORIGINAL WIDTH : 1024; ORIGINAL HEIGHT : 1024; TOP : 0; LEFT : 0; SCORE : 6.520; HPS: 3.220",
block_policy=2,
is_legacy=False,
use_cache=False,
)
image.save("output.webp")
