DALL·E Mini Model Card

Model Details

• Developed by: Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phúc Lê, Luke, Luke Melas, Ritobrata Ghosh
• Model type: Transformer-based text-to-image generation model
• Language(s): English
• License: Apache 2.0
• Model Description: This is a model that can be used to generate images based on text prompts. As the model developers wrote in the project report about DALL·E mini, “OpenAI had the first impressive model for generating images with DALL·E. DALL·E mini is an attempt at reproducing those results with an open-source model.”
• Resources for more information: See OpenAI’s website for more information about DALL·E, including the DALL·E model card. See the project report for more information from the model’s developers. To learn more about DALL·E Mega, see the DALL·E Mega training journal.
• Cite as:

Uses

Direct Use

Downstream Use

• Research efforts, such as probing and better understanding the limitations and biases of generative models to further improve the state of science
• Development of educational or creative tools
• Generation of artwork and use in design and artistic processes.
• Other uses that are newly discovered by users. This currently includes poetry illustration (give a poem as prompt), fan art (putting a character in various other visual universes), visual puns, fairy tale illustrations (give a fantasy situation as prompt), concept mashups (applying a texture to something completely different), style transfers (portraits in the style of), … We hope you will find your own application!

Misuse, Malicious Use, and Out-of-Scope Use

Out-of-Scope Use

Misuse and Malicious Use

• Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
• Intentionally promoting or propagating discriminatory content or harmful stereotypes.
• Impersonating individuals without their consent.
• Sexual content without consent of the people who might see it.
• Mis- and disinformation
• Representations of egregious violence and gore
• Sharing of copyrighted or licensed material in violation of its terms of use.
• Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.

Limitations and Bias

Limitations

• Faces and people in general are not generated properly.
• Animals are usually unrealistic.
• It is hard to predict where the model excels or falls short…Good prompt engineering will lead to the best results.
• The model has only been trained with English descriptions and will not perform as well in other languages

Bias

• Images generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
• When the model generates images with people in them, it tends to output people who we perceive to be white, while people of color are underrepresented.
• Images generated by the model can contain biased content that depicts power differentials between people of color and people who are white, with white people in positions of privilege.
• The model is generally only usable for generating images based on text in English, limiting accessibility of the model for non-English speakers and potentially contributing to the biases in images generated by the model.

Limitations and Bias Recommendations

• Users (both direct and downstream) should be made aware of the biases and limitations.
• Content that is potentially problematic should be filtered out, e.g., via automated models that detect violence or pornography.
• Further work on this model should include methods for balanced and just representations of people and cultures, for example, by curating the training dataset to be both diverse and inclusive.

Training

Training Data

• Conceptual Captions Dataset, which contains 3 million image and caption pairs.
• Conceptual 12M, which contains 12 million image and caption pairs.
• The OpenAI subset of YFCC100M, which contains about 15 million images and that we further sub-sampled to 2 million images due to limitations in storage space. They used both title and description as caption and removed html tags, new lines and extra spaces.

Training Procedure

• Images are encoded through a VQGAN encoder, which turns images into a sequence of tokens.
• Descriptions are encoded through a BART encoder.
• The output of the BART encoder and encoded images are fed through the BART decoder, which is an auto-regressive model whose goal is to predict the next token.
• Loss is the softmax cross-entropy between the model prediction logits and the actual image encodings from the VQGAN.

• Hardware: 1 pod TPU v3-256 = 32 nodes of TPU VM v3-8 (8 TPU per node) = 256 TPU v3
• Optimizer: Distributed Shampoo
• Model Partition Specificiations: 8 model parallel x 32 data parallel
• Batch: 44 samples per model x 32 data parallel x 3 gradient accumulation steps = 4224 increasing samples per update
• Learning rate: warmup to 0.0001 for 10,000 steps and then kept constant until plateau
• Gradient checkpointing used on each Encoder/Decoder layer (ie, MHA + FFN)
• Distributed Shampoo + Normformer Optimizations have proved to be effective and efficiently scaling this model.
• It should also be noted that the learning rate and other parameters are sometimes adjusted on the fly, and batch size increased over time as well.

Evaluation Results

Environmental Impact

DALL·E Mini Estimated Emissions

• Hardware Type: TPU v3-8
• Hours used: 72 (3 days)
• Cloud Provider: GCP (as mentioned in the technical report)
• Compute Region: us-east1 (provided by model developers)
• Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): 30.16 kg CO2 eq.

DALL·E Mega Estimated Emissions

• Hardware Type: TPU v3-256
• Hours used: 960 - 1080 hours (40-45 days)
• Cloud Provider: Unknown
• Compute Region: Unknown
• Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): Unknown

Citation