Model Card for pseudo-flex-base (1024x1024 base resolution)
img
stable-diffusion-2-1 (stabilityai/stable-diffusion-2-1) finetuned with different aspect ratios, into a photography model (ptx0/pseudo-real-beta).
Sample images
Seed: 2695929547
Steps: 25
Sampler: DDIM, default model config settings
Version: Pytorch 2.0.1, Diffusers 0.17.1
Guidance: 9.2
Guidance rescale: 0.0
resolution
model
stable diffusion
pseudo-flex
realism-engine
753x1004 (4:3)
v2-1
img
img
img
1280x720 (16:9)
v2-1
img
img
img
1024x1024 (1:1)
v2-1
img
img
img
1024x1024 (1:1)
v2-1
img
img
img
Background
The ptx0/pseudo-real-beta pretrained checkpoint had its unet trained for 4,200 steps and its text encoder trained for 15,600 steps at a batch size of 15 with 10 gradient accumulations, on a diverse dataset:
cushman (8000 kodachrome slides from 1939 to 1969)
national geographic (about 3-4,000 >1024x768 images of animals, wildlife, landscapes, history)
a small dataset of stock images of people vaping / smoking
It has a diverse capability of photorealistic and adventure with strong prompt coherence. However, it lacks multi-aspect capability.
The code used to train pseudo-real-beta did not have aspect bucketing support. I discovered pseudo-flex-base by @ttj, which supported theories I had.
Training code
I added thorough aspect bucketing support to my training loop dataloader by having it throw away any image under 1024x1024, and condition all images so that the smaller side of the image is 1024. The aspect ratio of the image is used to determine the new length of the other dimension, eg. used as a multiple for landscape or a divisor for portrait mode.
All batches have image of the same resolution. Different resolutions at the same aspect are all conditioned to 1024x... or ...x1024. A 1920x1080 image becomes approx 1820x1024.
Starting checkpoint
This model, pseudo-flex-base was created by fine-tuning the base stabilityai/stable-diffusion-2-1 768 model on its frozen text encoder, for 1000 steps on 148,000 images from LAION HD using the TEXT field as their caption.
The batch size was effectively 150 again. Batch size of 15 with 10 accumulations. This is very slow at very high resolutions, an aspect ratio of 1.5-1.7 will cause this to take about 700 seconds per iter on an A100 80G.
This training took two days.
Text encoder swap
At 1000 steps, the text encoder from ptx0/pseudo-real-beta was used experimentally with this model's unet in an attempt to resolve some residual image noise, eg. pixelation. That worked!
The training was restarted from ckpt 1000 with this text encoder.
The beginnings of wide / portrait aspect appearing
Validation prompts began to "pull together" from 1300 to 2950 steps. Some checkpoints show regression, but these usually resolve in about 100 steps. Improvements were always present, despite regresions.
Degradation and dataset swap
As training has been going on for some time now on 148,000 images at a batch size of 150 over 3000 steps, images began to degrade. This is presumably due to having completed 3 repeats on all images in the set, and that's IF all images in the set had been used. Considering some of the image filters discarded about 50,000 images, we landed at 9 repeats per image on our super low learning rate.
This caused two issues:
The images were beginning to show static noise.
The training was taking a very long time, and each checkpoint showed little improvement.
Overfitting to prompt vocabulary, and a lack of generalization.
Ergo, at 1300 steps, the decision was made to cease training on the original LAION HD dataset, and instead, train on a new freshly-retrieved subset of high-resolution Midjourney v5.1 data.
This consisted of 17,800 images at a base resolution of 1024x1024, with about 700 samples in portrait and 700 samples in landscape.
Contrast issues
As the checkpoint 3275 was tested, a common observation was that darker images were washed out, and brighter images seemed "meh".
Various CFG rescale and guidance levels were tested, with the best dark images occurring around guidance_scale=9.2 and guidance_rescale=0.0 but they remained "washed out".
Dataset change number two
A new LAION subset was prepared with unique images and no square images - just a limited collection of aspect ratios:
16:9
9:16
2:3
3:2
This was intended to speed up the understanding of the model, and prevent overfitting on captions.
This LAION subset contained 17,800 images, evenly distributed through aspect ratios.
The images were then captioned using T5 Flan with BLIP2, to obtain highly accurate results.
Contrast fix: offset noise / SNR gamma to the rescue?
Offset noise and SNR gamma were applied experimentally to the checkpoint 4250:
snr_gamma=5.0
noise_offset=0.2
noise_pertubation=0.1
Within 25 steps of training, the contrast was back, and the prompt a solid black square once again produced a reasonable result.
At 50 steps of offset noise, things really seemed to "click" and a solid black square had the fewest deformities I've seen.
Step 75 checkpoint was broken. The SNR gamma math results in numeric instability and was disabled. The offset noise parameters were untouched.
Success! Improvement in quality and contrast.
Similar to the text encoder swap, the images showed a marked improvement over the next several checkpoints.
It was left to its own devices, and at step 4475, enough improvement was observed that another revision in this repository was created.
Status: Test release
This model has been packaged up in a test form so that it can be thoroughly assessed by users.
stable-diffusion-2-1 (stabilityai/stable-diffusion-2-1 and ptx0/pseudo-real-beta) finetuned for dynamic aspect ratios.
finetuned resolutions:
width
height
aspect ratio
images
0
1024
1024
1:1
90561
1
1536
1024
3:2
8716
2
1365
1024
4:3
6933
3
1468
1024
~3:2
113
4
1778
1024
~5:3
6315
5
1200
1024
~5:4
6376
6
1333
1024
~4:3
2814
7
1281
1024
~5:4
52
8
1504
1024
~3:2
139
9
1479
1024
~3:2
25
10
1384
1024
~4:3
1676
11
1370
1024
~4:3
63
12
1499
1024
~3:2
436
13
1376
1024
~4:3
68
Other aspects were in smaller buckets. It could have been done more succinctly or carefully, but careless handling of the data was a part of the experiment parameters.
Developed by: pseudoterminal
Model type: Diffusion-based text-to-image generation model
All pre-processing is done via the scripts in bghira/SimpleTuner on GitHub.
Speeds, Sizes, Times
Dataset size: 100k image-caption pairs, after filtering.
Hardware: 1 A100 80G GPUs
Optimizer: 8bit Adam
Batch size: 150
actual batch size: 15
gradient_accumulation_steps: 10
effective batch size: 150
Learning rate: Constant 4e-8 which was adjusted by reducing batch size over time.
Training steps: WIP (ongoing)
Training time: approximately 4 days (so far)
Results
More information needed
Model Card Authors
pseudoterminal
How to Get Started with the Model
Use the code below to get started with the model.
python
1# Use Pytorch 2!2import torch
3from diffusers import StableDiffusionPipeline, DiffusionPipeline, AutoencoderKL, UNet2DConditionModel, DDPMScheduler
4from transformers import CLIPTextModel
56# Any model currently on Huggingface Hub.7model_id ='ptx0/pseudo-flex-base'8pipeline = DiffusionPipeline.from_pretrained(model_id)910# Optimize!11pipeline.unet = torch.compile(pipeline.unet)12scheduler = DDPMScheduler.from_pretrained(13 model_id,14 subfolder="scheduler"15)1617# Remove this if you get an error.18torch.set_float32_matmul_precision('high')1920pipeline.to('cuda')21prompts ={22"woman":"a woman, hanging out on the beach",23"man":"a man playing guitar in a park",24"lion":"Explore the ++majestic beauty++ of untamed ++lion prides++ as they roam the African plains --captivating expressions-- in the wildest national geographic adventure",25"child":"a child flying a kite on a sunny day",26"bear":"best quality ((bear)) in the swiss alps cinematic 8k highly detailed sharp focus intricate fur",27"alien":"an alien exploring the Mars surface",28"robot":"a robot serving coffee in a cafe",29"knight":"a knight protecting a castle",30"menn":"a group of smiling and happy men",31"bicycle":"a bicycle, on a mountainside, on a sunny day",32"cosmic":"cosmic entity, sitting in an impossible position, quantum reality, colours",33"wizard":"a mage wizard, bearded and gray hair, blue star hat with wand and mystical haze",34"wizarddd":"digital art, fantasy, portrait of an old wizard, detailed",35"macro":"a dramatic city-scape at sunset or sunrise",36"micro":"RNA and other molecular machinery of life",37"gecko":"a leopard gecko stalking a cricket"38}39for shortname, prompt in prompts.items():40# old prompt: ''41 image = pipeline(prompt=prompt,42 negative_prompt='malformed, disgusting, overexposed, washed-out',43 num_inference_steps=32, generator=torch.Generator(device='cuda').manual_seed(1641421826),44 width=1368, height=720, guidance_scale=7.5, guidance_rescale=0.3, num_inference_steps=25).images[0]45 image.save(f'test/{shortname}_nobetas.png',format="PNG")
