The surge in text-to-image models like Google's Imagen, OpenAI's DALL-E 3, and Stability AI's Stable Diffusion has revolutionized digital art creation. However, the effectiveness of Chinese text-to-image models, such as taiyi-diffusion-v0.1 and alt-diffusion based on SD v1.5, remains moderate. Many AI art platforms in China support only English or rely on Chinese-to-English translation tools. Current open-source text-to-image models predominantly support English, with limited bilingual capabilities. Our work, Taiyi-Diffusion-XL (Taiyi-XL), builds on these developments, focusing on enhancing Chinese text-to-image generation while retaining English proficiency, addressing the unique challenges of bilingual language processing.
The training of the Taiyi-Diffusion-XL text-to-image model encompasses three main stages. Initially, we created a high-quality dataset of image-text pairs, with each image accompanied by a detailed descriptive text. To overcome the limitations of web-crawled data, we employed advanced vision-language large models to generate accurate captions that precisely describe the images. This approach enriched our dataset, ensuring relevance and detail. Subsequently, we began with a pre-trained English CLIP model and expanded its vocabulary and position encoding to better support Chinese and longer texts. This expansion was achieved through training on a large-scale bilingual dataset, utilizing a contrastive loss function and a memory-efficient approach. Finally, based on Stable-Diffusion-XL, we replaced the text encoder obtained in the second stage and conducted multi-resolution, aspect-ratio-variant training of the diffusion model on the dataset prepared in the first stage. This comprehensive training process ensures that Taiyi-Diffusion-XL effectively supports bilingual text-to-image generation, catering to diverse linguistic and visual requirements.
Our machine evaluation involved a comprehensive comparison of various models. The evaluation metrics included CLIP Similarity (CLIP Sim), Inception Score (IS), and Fréchet Inception Distance (FID), providing a robust assessment of each model's performance in terms of image quality, diversity, and alignment with textual descriptions. In the English dataset (COCO), Taiyi-XL demonstrated superior performance across all metrics, achieving the highest scores in CLIP Sim, IS, and FID. This indicates Taiyi-XL's effectiveness in generating images closely aligned with English text prompts while maintaining high image quality and diversity. Similarly, in the Chinese dataset (COCO-CN), Taiyi-XL outperformed other models, showcasing its robust bilingual capabilities.
Table: Comparison of different models based on CLIP Sim, IS, and FID across English (COCO) and Chinese (COCO-CN) datasets
As shown in the figures below, a comparison of different models in Chinese and English text-to-image generation performance is presented. The XL version models, such as SD-XL and Taiyi-XL, show significant improvements over the 1.5 version models like SD-v1.5 and Alt-Diffusion. DALL-E 3 is renowned for its vibrant colors and its ability to closely follow text prompts, setting a high standard. Our Taiyi-XL model, with its photographic style, closely matches the performance of Midjourney and excels in bilingual (Chinese and English) text-to-image generation.
Although Taiyi-XL may not yet rival commercial models, it excels among current bilingual open-source models. The gap with commercial models is mainly due to differences in the quantity, quality, and diversity of training data. Our model is trained exclusively on copyright-compliant image-text data. We dont't use AI generated image such as Midjoueney or DALL-E 3. As is well known, copyright issues remain the biggest challenge in text-to-image and AI-generated content (AIGC) models.
We also evaluated the impact of using Latent Consistency Models (LCM) to accelerate the image generation process. The tests showed that as the number of inference steps decreases, the image quality declines. Extending the generation process to 8 steps generally ensures the quality of the generated images; when limited to a single step, the images mainly display basic outlines and lack finer details. This finding suggests that while LCM can effectively speed up the generation process, a balance must be struck between the number of steps and the desired image quality.
Taiyi-XL使用LCM - 中英文
引用 Citation
如果您在您的工作中使用了我们的模型，可以引用我们的论文：
If you are using the resource for your work, please cite the our paper:
text
1@misc{wu2024taiyidiffusionxl,
2 title={Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support},
3 author={Xiaojun Wu and Dixiang Zhang and Ruyi Gan and Junyu Lu and Ziwei Wu and Renliang Sun and Jiaxing Zhang and Pingjian Zhang and Yan Song},
4 year={2024},
5 eprint={2401.14688},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
text
1@article{fengshenbang,
2 author = {Jiaxing Zhang and Ruyi Gan and Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen},
3 title = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
4 journal = {CoRR},
5 volume = {abs/2209.02970},
6 year = {2022}
7}
