Model Overview
Description:
Cosmos AnomalyGen — Mobile Phone Screen (UC3) generates synthetic mobile-phone-screen anomaly images by inpainting a user-supplied binary mask onto a clean reference screen image, conditioned on one of three trained defect types (oil, scratch, stain). The release ships only the few-shot-finetuned modules — a set of anomaly-token embeddings and a 2-layer MLP adapter — which plug into the frozen Cosmos-Predict2 2B Text-to-Image diffusion backbone (also using a frozen NV-DINOv2 mask encoder and a frozen T5 text encoder) at inference time. Cosmos AnomalyGen — UC3 v1.0.0 was developed by NVIDIA as part of the Cosmos AnomalyGen pipeline. This model is ready for commercial use.
License/Terms of Use:
Governing Terms: Use of this model is governed by the
NVIDIA Open Model Agreement.
Deployment Geography:
Global
Use Case:
Industrial visual-inspection teams responsible for mobile-phone-screen QA who have very few (≤5 per defect type) real anomaly examples. The model produces large-scale synthetic anomaly datasets (clean screen + binary mask → realistic oil / scratch / stain image) for training downstream defect-detection or segmentation models, including downstream TAO toolkit consumers via the DAFT v3.0 export path.
Release Date:
References(s):
Model Architecture:
Architecture Type: Transformer (diffusion DiT backbone with learnable conditioning modules)
Network Architecture:
anomaly_embedding (trainable, included in this release): token embeddings (256 tokens per <texture>+<anomaly_type> pair) — three pairs trained for UC3: Phone+oil, Phone+scratch, Phone+stain.adapter (trainable, included in this release): 2-layer MLP with GELU activations (input / output hidden size = 1024), projecting the mask encoder output into the diffusion DiT conditioning space.mask_encoder (frozen, not redistributed in this release): NV-DINOv2 (ViT-L) backbone with adaptive pool (kernel = 7); weights are loaded from the separately downloaded NV-DINOv2 classification checkpoint at inference time.text_encoder (frozen, not redistributed in this release): google-t5/t5-large.- These modules condition the frozen Cosmos-Predict2 2B T2I DiT denoiser at inference time.
This model was developed based on Cosmos-Predict2-2B-Text2Image.
Number of model parameters: Approximately 2.9×10^6 (2.9 million) trainable parameters in the released modules — anomaly_embedding ≈ 0.79M (256 tokens × 1024 hidden × 3 <texture>+<anomaly_type> pairs) plus the 2-layer MLP adapter ≈ 2.1M (1024→1024 with GELU). The trainable modules are distributed as the model/iter_000009000.pt checkpoint file. The frozen Cosmos-Predict2 2B base contributes ~2.0×10^9 (2 billion) parameters used at inference time but not redistributed in this release.
Input(s):
Input Type(s): Image, Binary Mask, Text
Input Format(s):
- Image: PNG / JPG, Red, Green, Blue (RGB)
- Binary Mask: PNG / JPG, single-channel binary (0 = background, 255 = anomaly region; binarized at threshold 127)
- Text: anomaly-type string in the form
<texture>+<anomaly_type> (one of Phone+oil, Phone+scratch, Phone+stain)
Input Parameters:
- Image: Two-Dimensional (2D)
- Mask: Two-Dimensional (2D)
- Text: One-Dimensional (1D)
Other Properties Related to Input: Input clean image and paired mask must have the same dimensions; the model was trained at 512×512 and inference is run at the same resolution. anomaly_type must exactly match one of the three pairs trained for this UC3 checkpoint — passing an unsupported defect string is rejected by scripts/anomaly_gen/sdg-inference/validate_jsonl.py against this checkpoint's ag_config.yaml → dataloader_train.dataset.anomaly_types. The mask should ideally cover a contiguous defect region that resembles the trained mask distribution; the optional Automatic Mask Placement (AMP) tool can constrain placement to legal ROIs.
Output(s)
Output Type(s): Image
Output Format(s): PNG; Red, Green, Blue (RGB)
Output Parameters: Two-Dimensional (2D)
Other Properties Related to Output: 512×512 RGB synthetic anomaly image. Anomaly content is generated inside the user-supplied mask region; in the default crop_and_paste=True flow the inpainted patch is pasted back onto the clean reference image so non-masked pixels remain identical to the input. Optionally Poisson blending can be enabled. Generation metadata (per-sample guidance, crop_ratio, seed, etc.) is written to SDG_result.csv alongside the images.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration:
Runtime Engine(s):
- PyTorch (via the Cosmos-Predict2 2B T2I pipeline)
- Cosmos AnomalyGen scripts (
scripts.anomaly_gen.synthetic_dataset_generation, torchrun-based)
- NVIDIA TAO Toolkit — interop via DAFT v3.0 export (
scripts.anomaly_gen.convert_to_daft_format)
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere (A100)
- NVIDIA Hopper (H100)
- NVIDIA RTX 6000
Supported Operating System(s):
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
v1.0.0 — uc3-phone-screen-2b-512-iter9000 (trained 9,000 iterations; released artifact is the model/iter_000009000.pt checkpoint file containing finetuned modules only).
Training, Testing, and Evaluation Datasets:
Dataset Overview
- Total Number of Datasets: 2 (Roboflow
vu-thi-thu-huyen/mobile-screen + NGC cosmos-anomalygen-glass-dataset:1.0)
- Total Size: 15 anomaly RGB images + 15 paired binary masks + clean reference screen images (clean images are used at inference time, not as supervised training targets)
- Dataset partition: 100% Training (few-shot fine-tuning regime; no held-out validation or test split)
Public Datasets
vu-thi-thu-huyen/mobile-screen — Roboflow Universe, https://universe.roboflow.com/vu-thi-thu-huyen/mobile-screen — source of mobile-phone-screen RGB images (anomaly and clean).nv-metropolis-dev/metropolis-sdg/cosmos-anomalygen-glass-dataset:1.0 — NVIDIA NGC — source of binary defect masks and defect_spec.jsonl.
Training Dataset:
Data Modality:
Training Data Size:
- Less than a Million Images (15 anomaly images + 15 paired masks; 5 per defect type × 3 types)
** Data Collection Method by dataset
- Hybrid: Human, Automatic/Sensors
** Labeling Method by dataset
Properties: 15 RGB images of mobile phone screens spanning three defect categories — oil (5 images), scratch (5 images), stain (5 images) — each paired with a per-defect binary mask. Clean screen images from the same Roboflow source are used as inpainting references at inference time. No personal data, copyrighted content (other than the Roboflow CC-licensed source), or human subjects are present in the dataset.
Testing Dataset:
Not Applicable — the model is qualitatively evaluated via the log_image validation callback (inpainted samples logged every N training steps) rather than against a held-out test split.
Evaluation Dataset:
Not Applicable — no held-out evaluation dataset. The model is evaluated qualitatively during training via the
log_image validation callback, plus FID and nearest-neighbour metrics (
nn_score,
mnn_score) logged at each validation step. See the
Explainability subcard for performance metric details.
Inference:
Acceleration Engine: PyTorch (native, FP32). Multi-GPU rank-sharded inference is supported via torchrun --nproc_per_node=<N> with the predict2_anomaly_gen_fsdp_2b experiment.
Test Hardware:
- NVIDIA A100
- NVIDIA H100
- NVIDIA RTX 6000
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content.
Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
For more detailed information on ethical considerations for this model, please see the
Bias,
Explainability,
Privacy, and
Safety & Security subcards.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns
here.
Bias
| Field | Response |
|---|
| Participation considerations from adversely impacted groups protected classes in model design and testing: | Not Applicable — the model operates on images of mobile phone screens (inanimate objects) and does not target or classify human subjects. |
| Measures taken to mitigate against unwanted bias: | Not Applicable — the model is a synthetic anomaly image generator for industrial defect inspection and does not produce classifications or predictions on protected attributes. |
| Bias Metric (If Measured): | Not Measured. |
Explainability
| Field | Response |
|---|
| Intended Task/Domain: | Industrial visual inspection — synthetic anomaly image generation (image-to-image inpainting) for mobile phone screen defect detection. |
| Model Type: | Diffusion model — Transformer-based DiT denoiser (Cosmos-Predict2 2B, frozen) conditioned by learned anomaly-token embeddings, a frozen NV-DINOv2 mask encoder, and a trained 2-layer MLP adapter that projects mask-encoder features into the DiT conditioning space. |
| Intended Users: | ML engineers and computer-vision practitioners building defect-detection or segmentation models for mobile phone screen QA. |
| Output: | Image (single 512×512 RGB anomaly image per inference call; multiple per testcase line when num_generated_images > 1). |
| Describe how the model works: | A user-supplied binary mask indicating where the defect should appear is encoded by a frozen NV-DINOv2 backbone and projected through a trained 2-layer MLP adapter into the diffusion DiT conditioning space. The selected anomaly-type token (Phone+oil, Phone+scratch, or Phone+stain) retrieves a learned 256-token embedding which further conditions the frozen Cosmos-Predict2 2B T2I DiT denoiser. The denoiser then produces an inpainting-style sample where pixels inside the mask are generated as a defect that matches the trained type, while pixels outside the mask are preserved from the clean reference image (with optional crop-and-paste / Poisson blending). |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable |
| Technical Limitations & Mitigation: | Trained on only 5 anomaly examples per defect type. Out-of-distribution screen backgrounds, mask placements that deviate substantially from the trained distribution, or any defect type outside {oil, scratch, stain} may yield low-fidelity outputs. Mitigation: (a) use the Automatic Mask Placement (AMP) tool, which automatically restricts the user-supplied mask to plausible defect regions on the mobile phone screen, preventing the model from generating defects in implausible locations; (b) run scripts/anomaly_gen/filter.py on the generated dataset — the filter scores each output with a Generative Image Quality Assessment (G-IQA) model and discards samples whose quality/realism score falls below a configurable threshold, so unrealistic synthetic images do not pollute the downstream training set; (c) validate any downstream detector trained on synthetic samples against real defect imagery before deployment. |
| Verified to have met prescribed NVIDIA quality standards: | Yes. |
| Performance Metrics: | FID (logged during training validation), nearest-neighbor metrics (nn_score, mnn_score), and visual inspection of log_image callback outputs. |
| Potential Known Risks: | Generated synthetic defects may not perfectly cover all real-world defect variations. Downstream detection models trained primarily on these synthetic samples should be validated against real defect data before deployment in production QA lines.
This model can generate synthetic images and may produce content that is offensive, unsafe, misleading, indecent, or unsuitable for a target deployment. Users should implement robust safety guardrails — including content filtering, abuse monitoring, and access controls — to reduce the risk of harmful outputs. Users are responsible for ensuring that their use of the model complies with all applicable laws and regulations, and for regularly reviewing and updating their guardrails as risks evolve. |
| Licensing: | Governing Terms: Use of this model is governed by the NVIDIA Open Model Agreement.
|
Privacy
| Field | Response |
|---|
| Generatable or reverse engineerable personal data? | No |
| Personal data used to create this model? | Unknown (Externally-Sourced) |
| Was consent obtained for any personal data used? | Not Applicable |
| How often is dataset reviewed? | Before Release |
| Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model? | No |
| Is there provenance for all datasets used in training? | Yes |
| Does data labeling (annotation, metadata) comply with privacy laws? | Yes |
| Is data compliant with data subject requests for data correction or removal, if such a request was made? | Not Applicable |
| Applicable Privacy Policy | https://www.nvidia.com/en-us/about-nvidia/privacy-policy/ |
Safety
| Field | Response |
|---|
| Model Application Field(s): | Industrial/Machinery and Robotics — specifically synthetic data generation for manufacturing QA / visual inspection of mobile phone screens. |
| Describe the life critical impact (if present). | Not Applicable. |
| Use Case Restrictions: | Governing Terms: Use of this model is governed by the NVIDIA Open Model Agreement.
|
| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
