Discovering and Mitigating Visual Biases through Keyword Explanation
arxiv(2023)
摘要
Addressing biases in computer vision models is crucial for real-world AI
deployments. However, mitigating visual biases is challenging due to their
unexplainable nature, often identified indirectly through visualization or
sample statistics, which necessitates additional human supervision for
interpretation. To tackle this issue, we propose the Bias-to-Text (B2T)
framework, which interprets visual biases as keywords. Specifically, we extract
common keywords from the captions of mispredicted images to identify potential
biases in the model. We then validate these keywords by measuring their
similarity to the mispredicted images using a vision-language scoring model.
The keyword explanation form of visual bias offers several advantages, such as
a clear group naming for bias discovery and a natural extension for debiasing
using these group names. Our experiments demonstrate that B2T can identify
known biases, such as gender bias in CelebA, background bias in Waterbirds, and
distribution shifts in ImageNet-R/C. Additionally, B2T uncovers novel biases in
larger datasets, such as Dollar Street and ImageNet. For example, we discovered
a contextual bias between "bee" and "flower" in ImageNet. We also highlight
various applications of B2T keywords, including debiased training, CLIP
prompting, and model comparison.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要