VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
arxiv(2024)
摘要
Visual entailment (VE) is a multimodal reasoning task consisting of
image-sentence pairs whereby a promise is defined by an image, and a hypothesis
is described by a sentence. The goal is to predict whether the image
semantically entails the sentence. VE systems have been widely adopted in many
downstream tasks. Metamorphic testing is the commonest technique for AI
algorithms, but it poses a significant challenge for VE testing. They either
only consider perturbations on single modality which would result in
ineffective tests due to the destruction of the relationship of image-text
pair, or just conduct shallow perturbations on the inputs which can hardly
detect the decision error made by VE systems. Motivated by the fact that
objects in the image are the fundamental element for reasoning, we propose
VEglue, an object-aligned joint erasing approach for VE systems testing. It
first aligns the object regions in the premise and object descriptions in the
hypothesis to identify linked and un-linked objects. Then, based on the
alignment information, three Metamorphic Relations are designed to jointly
erase the objects of the two modalities. We evaluate VEglue on four widely-used
VE systems involving two public datasets. Results show that VEglue could detect
11,609 issues on average, which is 194
addition, VEglue could reach 52.5
significantly outperform the baselines by 17.1
the tests generated by VEglue to retrain the VE systems, which largely improves
model performance (50.8
sacrificing the accuracy on the original test set.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要