Stable Diffusion 3.5 Medium is a Multimodal Diffusion Transformer with improvements (MMDiT-X) text-to-image model that features improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.
Please note: This model is released under the
Stability Community License. Visit
Stability AI to learn or
contact us for commercial licensing details.
For local or self-hosted use, we recommend
ComfyUI for node-based UI inference, or
diffusers or
GitHub for programmatic use.
-
MMDiT-X: Introduces self-attention modules in the first 13 layers of the transformer, enhancing multi-resolution generation and overall image coherence.
-
QK Normalization: Implements the QK normalization technique to improve training Stability.
-
Mixed-Resolution Training:
- Progressive training stages: 256 → 512 → 768 → 1024 → 1440 resolution
- The final stage included mixed-scale image training to boost multi-resolution generation performance
- Extended positional embedding space to 384x384 (latent) at lower resolution stages
- Employed random crop augmentation on positional embeddings to enhance transformer layer robustness across the entire range of mixed resolutions and aspect ratios. For example, given a 64x64 latent image, we add a randomly cropped 64x64 embedding from the 192x192 embedding space during training as the input to the x stream.
These enhancements collectively contribute to the model's improved performance in multi-resolution image generation, coherence, and adaptability across various text-to-image tasks.
-
Text Encoders：
- CLIPs: OpenCLIP-ViT/G, CLIP-ViT/L, context length 77 tokens
- T5: T5-xxl, context length 77/256 tokens at different stages of training
-
Training Data and Strategy:
This model was trained on a wide variety of data, including synthetic data and filtered publicly available data.
For more technical details of the original MMDiT architecture, please refer to the
Research paper.
See
blog for our study about comparative performance in prompt adherence and aesthetic quality.
1├── text_encoders/
2│ ├── README.md
3│ ├── clip_g.safetensors
4│ ├── clip_l.safetensors
5│ ├── t5xxl_fp16.safetensors
6│ └── t5xxl_fp8_e4m3fn.safetensors
7│
8├── README.md
9├── LICENSE
10├── sd3.5_medium.safetensors
11├── SD3.5M_example_workflow.json
12├── SD3.5M_SLG_example_workflow.json
13├── SD3.5L_plus_SD3.5M_upscaling_example_workflow.json
14└── sd3_medium_demo.jpg
15
16** File structure below is for diffusers integration**
17├── scheduler/
18├── text_encoder/
19├── text_encoder_2/
20├── text_encoder_3/
21├── tokenizer/
22├── tokenizer_2/
23├── tokenizer_3/
24├── transformer/
25├── vae/
26└── model_index.json
Upgrade to the latest version of the
🧨 diffusers library1import torch
2from diffusers import StableDiffusion3Pipeline
3
4pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3.5-medium", torch_dtype=torch.bfloat16)
5pipe = pipe.to("cuda")
6
7image = pipe(
8 "A capybara holding a sign that reads Hello World",
9 num_inference_steps=40,
10 guidance_scale=4.5,
11).images[0]
12image.save("capybara.png")
1from diffusers import BitsAndBytesConfig, SD3Transformer2DModel
2from diffusers import StableDiffusion3Pipeline
3import torch
4
5model_id = "stabilityai/stable-diffusion-3.5-medium"
6
7nf4_config = BitsAndBytesConfig(
8 load_in_4bit=True,
9 bnb_4bit_quant_type="nf4",
10 bnb_4bit_compute_dtype=torch.bfloat16
11)
12model_nf4 = SD3Transformer2DModel.from_pretrained(
13 model_id,
14 subfolder="transformer",
15 quantization_config=nf4_config,
16 torch_dtype=torch.bfloat16
17)
18
19pipeline = StableDiffusion3Pipeline.from_pretrained(
20 model_id,
21 transformer=model_nf4,
22 torch_dtype=torch.bfloat16
23)
24pipeline.enable_model_cpu_offload()
25
26prompt = "A whimsical and creative image depicting a hybrid creature that is a mix of a waffle and a hippopotamus, basking in a river of melted butter amidst a breakfast-themed landscape. It features the distinctive, bulky body shape of a hippo. However, instead of the usual grey skin, the creature's body resembles a golden-brown, crispy waffle fresh off the griddle. The skin is textured with the familiar grid pattern of a waffle, each square filled with a glistening sheen of syrup. The environment combines the natural habitat of a hippo with elements of a breakfast table setting, a river of warm, melted butter, with oversized utensils or plates peeking out from the lush, pancake-like foliage in the background, a towering pepper mill standing in for a tree. As the sun rises in this fantastical world, it casts a warm, buttery glow over the scene. The creature, content in its butter river, lets out a yawn. Nearby, a flock of birds take flight"
27
28image = pipeline(
29 prompt=prompt,
30 num_inference_steps=40,
31 guidance_scale=4.5,
32 max_sequence_length=512,
33).images[0]
34image.save("whimsical.png")
Please see the fine-tuning guide
here.
All uses of the model must be in accordance with our
Acceptable Use Policy.
The model was not trained to be factual or true representations of people or events. As such, using the model to generate such content is out-of-scope of the abilities of this model.
As part of our safety-by-design and responsible AI deployment approach, we take deliberate measures to ensure Integrity starts at the early stages of development. We implement safety measures throughout the development of our models. We have implemented safety mitigations that are intended to reduce the risk of certain harms, however we recommend that developers conduct their own testing and apply additional mitigations based on their specific use cases.
For more about our approach to Safety, please visit our
Safety page.
Our integrity evaluation methods include structured evaluations and red-teaming testing for certain harms. Testing was conducted primarily in English and may not cover all possible harms.
