Stable Diffusion v2-1-base Model Card

coreml-stable-diffusion-2-1-base
├── original
│   ├── compiled              # Swift inference, "original" attention
│   └── packages              # Python inference, "original" attention
└── split_einsum
    ├── compiled              # Swift inference, "split_einsum" attention
    └── packages              # Python inference, "split_einsum" attention

• coreml-stable-diffusion-2-1-base-palettized_original_compiled.zip contains the compiled, 6-bit model with ORIGINAL attention implementation.
• coreml-stable-diffusion-2-1-base-palettized_split_einsum_v2_compiled.zip contains the compiled, 6-bit model with SPLIT_EINSUM_V2 attention implementation.

• Use it with 🧨 diffusers
• Use it with the stablediffusion repository: download the v2-1_512-ema-pruned.ckpt here.

Model Details

• Developed by: Robin Rombach, Patrick Esser
• Model type: Diffusion-based text-to-image generation model
• Language(s): English
• License: CreativeML Open RAIL++-M License
• Model Description: This is a model that can be used to generate and modify images based on text prompts. It is a Latent Diffusion Model that uses a fixed, pretrained text encoder (OpenCLIP-ViT/H).
• Resources for more information: GitHub Repository.
• Cite as:

  @InProceedings{Rombach_2022_CVPR,
      author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
      title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2022},
      pages     = {10684-10695}
  }

Uses

Direct Use

• Safe deployment of models which have the potential to generate harmful content.
• Probing and understanding the limitations and biases of generative models.
• Generation of artworks and use in design and other artistic processes.
• Applications in educational or creative tools.
• Research on generative models.

Misuse, Malicious Use, and Out-of-Scope Use

Out-of-Scope Use

Misuse and Malicious Use

• Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
• Intentionally promoting or propagating discriminatory content or harmful stereotypes.
• Impersonating individuals without their consent.
• Sexual content without consent of the people who might see it.
• Mis- and disinformation
• Representations of egregious violence and gore
• Sharing of copyrighted or licensed material in violation of its terms of use.
• Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.

Limitations and Bias

Limitations

• The model does not achieve perfect photorealism
• The model cannot render legible text
• The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
• Faces and people in general may not be generated properly.
• The model was trained mainly with English captions and will not work as well in other languages.
• The autoencoding part of the model is lossy
• The model was trained on a subset of the large-scale dataset LAION-5B, which contains adult, violent and sexual content. To partially mitigate this, we have filtered the dataset using LAION's NFSW detector (see Training section).

Bias

Training

• LAION-5B and subsets (details below). The training data is further filtered using LAION's NSFW detector, with a "p_unsafe" score of 0.1 (conservative). For more details, please refer to LAION-5B's NeurIPS 2022 paper and reviewer discussions on the topic.

• Images are encoded through an encoder, which turns images into latent representations. The autoencoder uses a relative downsampling factor of 8 and maps images of shape H x W x 3 to latents of shape H/f x W/f x 4
• Text prompts are encoded through the OpenCLIP-ViT/H text-encoder.
• The output of the text encoder is fed into the UNet backbone of the latent diffusion model via cross-attention.
• The loss is a reconstruction objective between the noise that was added to the latent and the prediction made by the UNet. We also use the so-called v-objective, see https://arxiv.org/abs/2202.00512.

Version 2.1

• 512-base-ema.ckpt: Fine-tuned on 512-base-ema.ckpt 2.0 with 220k extra steps taken, with punsafe=0.98 on the same dataset.
• 768-v-ema.ckpt: Resumed from 768-v-ema.ckpt 2.0 with an additional 55k steps on the same dataset (punsafe=0.1), and then fine-tuned for another 155k extra steps with punsafe=0.98.

Version 2.0

• 512-base-ema.ckpt: 550k steps at resolution 256x256 on a subset of LAION-5B filtered for explicit pornographic material, using the LAION-NSFW classifier with punsafe=0.1 and an aesthetic score >= 4.5. 850k steps at resolution 512x512 on the same dataset with resolution >= 512x512.
• 768-v-ema.ckpt: Resumed from 512-base-ema.ckpt and trained for 150k steps using a v-objective on the same dataset. Resumed for another 140k steps on a 768x768 subset of our dataset.
• 512-depth-ema.ckpt: Resumed from 512-base-ema.ckpt and finetuned for 200k steps. Added an extra input channel to process the (relative) depth prediction produced by MiDaS (dpt_hybrid) which is used as an additional conditioning. The additional input channels of the U-Net which process this extra information were zero-initialized.
• 512-inpainting-ema.ckpt: Resumed from 512-base-ema.ckpt and trained for another 200k steps. Follows the mask-generation strategy presented in LAMA which, in combination with the latent VAE representations of the masked image, are used as an additional conditioning. The additional input channels of the U-Net which process this extra information were zero-initialized. The same strategy was used to train the 1.5-inpainting checkpoint.
• x4-upscaling-ema.ckpt: Trained for 1.25M steps on a 10M subset of LAION containing images >2048x2048. The model was trained on crops of size 512x512 and is a text-guided latent upscaling diffusion model. In addition to the textual input, it receives a noise_level as an input parameter, which can be used to add noise to the low-resolution input according to a predefined diffusion schedule.
• Hardware: 32 x 8 x A100 GPUs
• Optimizer: AdamW
• Gradient Accumulations: 1
• Batch: 32 x 8 x 2 x 4 = 2048
• Learning rate: warmup to 0.0001 for 10,000 steps and then kept constant

Evaluation Results

Environmental Impact

• Hardware Type: A100 PCIe 40GB
• Hours used: 200000
• Cloud Provider: AWS
• Compute Region: US-east
• Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): 15000 kg CO2 eq.

Citation

@InProceedings{Rombach_2022_CVPR,
    author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
    title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {10684-10695}
}