Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support
CoRR(2024)
摘要
Recent advancements in text-to-image models have significantly enhanced image
generation capabilities, yet a notable gap of open-source models persists in
bilingual or Chinese language support. To address this need, we present
Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
which is developed by extending the capabilities of CLIP and
Stable-Diffusion-XL through a process of bilingual continuous pre-training.
This approach includes the efficient expansion of vocabulary by integrating the
most frequently used Chinese characters into CLIP's tokenizer and embedding
layers, coupled with an absolute position encoding expansion. Additionally, we
enrich text prompts by large vision-language model, leading to better images
captions and possess higher visual quality. These enhancements are subsequently
applied to downstream text-to-image models. Our empirical results indicate that
the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
previous models. This research leads to the development and open-sourcing of
the Taiyi-Diffusion-XL model, representing a notable advancement in the field
of image generation, particularly for Chinese language applications. This
contribution is a step forward in addressing the need for more diverse language
support in multimodal research. The model and demonstration are made publicly
available at
\href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{this
https URL}, fostering further research and collaboration in this domain.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要