Holistic Evaluation of Text-To-Image Models
NeurIPS(2023)
摘要
The stunning qualitative improvement of recent text-to-image models has led
to their widespread attention and adoption. However, we lack a comprehensive
quantitative understanding of their capabilities and risks. To fill this gap,
we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
(HEIM). Whereas previous evaluations focus mostly on text-image alignment and
image quality, we identify 12 aspects, including text-image alignment, image
quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
encompassing these aspects and evaluate 26 state-of-the-art text-to-image
models on this benchmark. Our results reveal that no single model excels in all
aspects, with different models demonstrating different strengths. We release
the generated images and human evaluation results for full transparency at
https://crfm.stanford.edu/heim/v1.1.0 and the code at
https://github.com/stanford-crfm/helm, which is integrated with the HELM
codebase.
更多查看译文
关键词
models,evaluation,text-to-image
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要