The standard CLIP cannot generate anything but noise, so be sure to use the accompanying CLIP. "CLIP_INCL" file includes both the DiT and CLIP models. T5XXL has not been modified, so you can use the standard version.
Additionally, during inference, set the model shift to around 7; otherwise, the generated images will appear overburned. In the case of ComfyUI, you can adjust the values using the ModelSamplingSD3 Node.
My English is terrible, so I use translation tools.
Description
This is an experimental anime model designed to explore training methods for the flow-matching DiT model, SD3.5 Medium.
Since the model is still in the training phase, the art style lacks consistency, and there are many issues with hands and body structures.
Due to the small amount of training, the <character><series><artists> tags are almost non-functional. As training is focused on girl characters, it may not generate boy or other non-persons well. Since the dataset was created using hakubooru, the prompt format will be the same as the KohakuXL format. However, based on experiments, it is not strictly necessary to follow this format, as it interprets meaning to some extent even in natural language.
Special Tags
Quality Tags: masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality
Rating Tags: safe, sensitive, nsfw, explicit
Date Tags: newest, recent, mid, early, old
Training
Observations and Reflections
Insights and considerations gained through training.
Note: These reflections are based on training with anime images, and may not apply to photo images.
Curriculum Learning: From early experiments, I realized that the DiT model (or flow-matching models) learn concepts differently compared to conventional diffusion models. Therefore, I adopted a multi-stage training flow, starting with low-frequency components such as composition and poses to build a stable foundation, followed by learning high-frequency details.
Gradual Resolution Increase: As comprehensive training from the beginning proved challenging, I first trained at a lower resolution to grasp basic concepts before transitioning to higher resolutions. Training was conducted in two stages: 512px → 1024px. Incorporating low-resolution training also helped reduce training time. The enable_scaled_pos_embed option caused increased artifacts outside the training resolution in my tests, so I trained without using it.
T5 Attention Mask: Comparing results with and without the mask, I found that training without the mask produced more natural results. Hence, the mask was not used.
Weighting Scheme: Under my conditions, using a uniform weighting scheme led to increasing breakdowns in composition as large-scale training progressed. I found that for large-scale training, logit_normal (which strongly emphasizes subjects like people) or mode (which learns the entire image, including the background) worked better.
Training Shift: The SD3 technical report and official implementation use a default shift value of 3.0. However, using shift=3.0 at 1024px resolution often caused significant anatomical issues. Based on my tests, shift=2.5 worked well at 512px, while shift=5.0 or 7.0 was a balanced parameter for 1024px, minimizing artifacts and capturing details effectively. With shift values of 10 or 20, low-frequency regions were emphasized, leading to stronger learning of the background. Since higher shifts reduce the number of generative steps, I recommend a shift of around 6 for inference.
Optimizers: I found AdamW, ScheduleFreeAdamW, and ADOPT to be effective. Among these, Cautious ADOPT, which combines the Cautious optimizer, was particularly effective. However, it could become noisy depending on the dataset, so careful usage is required.
Batch Size: The high batch sizes commonly used for full fine-tuning models amplified instability in the model. Therefore, significantly lower batch sizes, such as 4 or 8, allowed for more stable training.
Local Minima: During training, issues such as parts of the image turning black or outputs becoming unusually dark were observed. These issues often occurred when the learning rate was low, suggesting the model was trapped in local minima. Increasing the learning rate (e.g., to 7.5e-4) showed a tendency to resolve these problems.
training_shift: SD3の技術レポートや公式実装ではshift=3.0がデフォルトで使用されています。しかしshift=3.0は解像度1024で使用すると人体の破綻がかなりの割合で発生します。私が検証した限りでは、解像度512でshift=2.5、解像度1024ではshift=5.0 or 7.0が破綻しづらく、かつ細部も表現できるバランスのよいパラメータであるとわかりました。shift=10,20になると低周波領域が強調され背景が強く学習されます。生成ステップが減少するので少なくとも生成時のshiftは6前後をお勧めします。
