This is a Japanese input support version of the image generation model SDXL with a total of 5.8B parameters. Here, we release the Japanese input support version of the base model (stable-diffusion-xl-base-1.0).
Training Strategy
Fine-tuning
We fine-tuned only the text encoders used in stable-diffusion-xl-base-1.0, OpenCLIP-ViT/G, CLIP-ViT/L to support Japanese input. We used Japanese-English parallel corpus as training dataset. We trained the Japanese text encoder so that the output (hidden states) when English sentences were input to the original English text encoder and the output when the same meaning Japanese sentences were input to the newly trained Japanese text encoder were the same. We used line-corporation/japanese-large-lm-3.6b as Japanese tokenizer.
We trained a text encoder that supports Japanese input by fine-tuning only the text encoders used in stable-diffusion-xl-base-1.0, OpenCLIP-ViT/G and CLIP-ViT/L. Specifically, we trained the new Japanese text encoder to produce output that matches the output (hidden states) of the original text encoder when the same meaning Japanese sentences and English sentences are input. We used Japanese-English parallel data as the training data and employed the line-corporation/japanese-large-lm-3.6b as the Japanese tokenizer.
A large-scale dataset consisting of image-caption pairs released by Christoph et al. (2022). We used only the captions for training this model. As a preprocessing step, we filtered the Japanese captions using fasttext, and then used the top 13221368 captions with high similarity to the images. We used rinna/japanese-cloob-vit-b-16 to calculate the similarity between images and captions. We translated the Japanese captions into English captions using a Japanese-English translation model, ABCI-baee, used in NT5 at WMT 2022 General Translation Task.
