Controls image generation by edge maps generated with Edge Drawing. Note that Edge Drawing comes in different flavors: original (ed), parameter free (edpf), color (edcolor).
canny 1.0 model was trained on 3M images with fp32, canny 1.1 model on even more, while edpf model so far is only trained on a 180k-360k with fp16.
canny edge-detector requires parameter tuning while edpf is parameter free.
Should we manually fine-tune canny to find the perfect input image or do we leave it at default? We could argue that "no fine-tuning required" is the usp of edpf and we want to compare in the default setting, whereas canny fine-tuning is subjective.
Would the canny model actually benefit from a edpf pre-processor and we might not even require a specialized edpf model? (2023-09-25: see eval_canny_edpf.zip but it seems as if it doesn't work and the edpf model may be justified)
When evaluating human images we need to be aware of Stable Diffusion's inherent limits, like disformed faces and hands, and don't attribute them to the control net.
When evaluating style we need to be aware of the bias from the image dataset (laion2b-en-aesthetics65), which might tend to generating "aesthetic" images, and not actually work "intrisicly better".
Trained for 40000 steps with default settings => results are not good. empty prompts were probably too excessive. retry with no drops and different algorithm parameters.
resumed with epoch 2 from 90000 using --proportion_empty_prompts=0.5 => results became worse, CN didn't pick up on no-prompts (I also tried intermediate checkpoint-104000). restarting with 50% drop.
see experiment 3.0. restarted from 0 with --proportion_empty_prompts=0.5 => results are not good, 50% is probably too much for 45k steps. guessmode still doesn't work and tends to produces humans. resuming until 90k with right-left flipped in the hope it will get better with more images.
resumed from 45000 steps with left-right flipped images until 90000 steps => results are still not good, 50% is probably also too much for 90k steps. guessmode still doesn't work and tends to produces humans. aborting.
resized images with shortside to 512 which gives us rectangular images instead of 512x512 squares
included images with aspect ratio > 2
center-cropped images to 512x(n)*64 | n=8..16 , which keeps them SD compatible
sorted duplicates by similarity value from laion2b-en-aesthetics65 to get the "best" text from all the duplicates according to laion
183410 images in total
75686 duplicates
381 outliers
50 too dark
436 too bright
31 blurry
76288 unique removed (that's 42%!)
------
107122 unique images (x2 left-right flip)
1 epoch = 107122 * 2 / 4 = 53561 steps per epoch
restarted from 0 and --mixed-precision="fp16".
TODO: Why did I end up with less images after I added more images? fastdup suddenly finds even more duplicates. Is fastdup default threshold=0.9 too aggressive?
see experiment 6.0. restarted from 0 with --train_batch_size=2 --gradient_accumulation_steps=16. 1 epoch = 107122 * 2 / 32 = 6696 steps per epoch => released as version 0.2 on civitai.
