2023-01-02: I wasted more GPU hours to train it a little bit more overfitting. Check out bp_mk3.safetensors and bp_mk5.safetensors. Prepare yourself own VAE! Update your WebUI if you can't load safetensors. Adds lots of samples in images folder!

2023-01-06: Checkout NMFSAN for a new model trained with custom embeddings.

BPModel is an experimental Stable Diffusion model based on ACertainty from Joseph Cheung.

Why is the Model even existing? There are loads of Stable Diffusion model out there, especially anime style models. Well, is there any models trained with resolution base resolution (base_res) 768 even 1024 before? Don't think so. Here it is, the BPModel, a Stable Diffusion model you may love or hate. Trained with 5k high quality images that suit my taste (not necessary yours unfortunately) from Sankaku Complex with annotations. The dataset is public in Crosstyan/BPDataset for the sake of full disclosure . Pure combination of tags may not be the optimal way to describe the image, but I don't need to do extra work. And no, I won't feed any AI generated image to the model even it might outlaw the model from being used in some countries.

The training of a high resolution model requires a significant amount of GPU hours and can be costly. In this particular case, 10 V100 GPU hours were spent on training 30 epochs with a resolution of 512, while 60 V100 GPU hours were spent on training 30 epochs with a resolution of 768. An additional 100 V100 GPU hours were also spent on training a model with a resolution of 1024, although ONLY 10 epochs were run. The results of the training on the 1024 resolution model did not show a significant improvement compared to the 768 resolution model, and the resource demands, achieving a batch size of 1 on a V100 with 32G VRAM, were high. However, training on the 768 resolution did yield better results than training on the 512 resolution, and it is worth considering as an option. It is worth noting that Stable Diffusion 2.x also chose to train on a 768 resolution model. However, it may be more efficient to start with training on a 512 resolution model due to the slower training process and the need for additional prior knowledge to speed up the training process when working with a 768 resolution.

Mikubill/naifu-diffusion is used as training script and I also recommend to checkout CCRcmcpe/scal-sdt.

The configuration for 1024 and 768 resolution with aspect ratio bucket is presented here.

The limitation described in SCAL-SDT Wiki is still applied.

BPModel can generate more proper kitty cat (if you know what I mean) than other anime model, but it's still not perfect. As results presented in Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models, the copy and paste effect is still observed.

Anything v3™ has been proven to be the most popular anime model in the community, but it's not perfect either as described in JosephusCheung/ACertainThing

BPModel, which has been fine-tuned on a relatively small dataset, is prone to overfit inherently. This is not surprising given the size of the dataset, but the strong prior knowledge of ACertainty (full Danbooru) and Stable Diffusion (LAION) helps to minimize the impact of overfitting. However I believe it would perform better than some artist style DreamBooth model which only train with a few hundred images or even less. I also oppose changing style by merging model since You could apply different style by training with proper captions and prompting.

Besides some of images in my dataset have the artist name in the caption, however some artist name will be misinterpreted by CLIP when tokenizing. For example, as109 will be tokenized as [as, 1, 0, 9] and fuzichoco will become [fu, z, ic, hoco]. Romanized Japanese suffers from the problem a lot and I don't have a good solution to fix it other than changing the artist name in the caption, which is time consuming and you can't promise the token you choose is unique enough. Remember the sks?

Language drift problem is still exist. There's nothing I can do unless I can find a way to generate better caption or caption the image manually. OFA combined with convnext-tagger could provide a better result for SFW content. However fine tune is necessary for NSFW content, which I don't think anyone would like to do. (Could Unstable Diffusion give us surprise?)

Here're some cherry picked samples.

I were using xformers when generating these sample and it might yield slight different result even with the same seed (welcome to the non deterministic field). "Upscale latent space image when doing hires. fix" is enabled also.

more samples can be found in images folder.

The bp_1024_e10.ckpt doesn't include any VAE and you should using other popular VAE in the community when using with AUTOMATIC1111/stable-diffusion-webui or you would see the LaTeNt SpAcE!

Use bp_1024_with_vae_te.ckpt if you don't have VAE and text encoder with you, still EMA weight is not included and it's fp16.

If you want to continue training, use bp_1024_e10_ema.ckpt which is the ema unet weight and with fp32 precision.

For better performance, it is strongly recommended to use Clip skip (CLIP stop at last layers) 2. It's also recommended to use turn on "Upscale latent space image when doing hires. fix" in the settings of AUTOMATIC1111/stable-diffusion-webui which adds intricate details when using Highres. fix.

I asked the chatGPT what the proper explanation of abbreviation BP could be.

Personally, I would call it "Big Pot".

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage. The CreativeML OpenRAIL License specifies: