Twisted-Reality-SANA-4.8B
This is a finetune of the SANA 4.8B model. The dataset consists primarly of Playboy photos, hollywood stars and National Geographics.
The model has been trained for research only, and shall not be used for commercial use.
Image galery
Intent
The intent of this model was to explore the limitations of doing diffusion under an heavily compressed latent space.
Some observations:
- Perturbed Attention Guidance (PAG) is essential to smooth out the latent space, at the expense of globally simpler image composition.
- The base checkpoint is severly undertrained for human anatomy, especially the 4.8B model as it seemed to not exploit all its weights.
- The model is excellent at closeup photo portraits, but weaker for full body shots, i.e., the faces get smudged at farther distances.
- Training using full weights was mandatory, as finetuning using a PEFT method didn't make the base model drift away enough to learn correct anatomy.
- The "prompt augmentation chi" method doesn't seem to have any effect on the prompt, something to be investigated.
I've also explored adding some residual paths for each of the transformer block, but it ended up as degrading the image quality, while making it easier to train.
I ended up switching back to the base checkpoint architecture for this reason.
Training hyperparameters
The model has been trained in multiple sessions, but these parameters gave the final version (and best results):
- Batch Size = 8 on 4x A100
- Gradient Accumulation = 8 steps
- Learning rate = 1e-4 constant
- Scheduler : AdamW
Possible improvements
Currently, the only improvement I see is to, unfortunately, further increase the number of parameters of the transformer.
Indeed, there is a significant increase of observable image quality from 1.6B parameters to 4.8B parameters.
Increasing the MLP ratios, currently at 2.5 could help, but this results in increasing the total number of parameters.
There are other possible changes, such as introducing sincos positional embedding, full self attention mechanism, but these are not proven to improve the image quality.
Future development
Future development could involve further training using a lower learning rate.
