Learning from Models and Data for Visual Grounding
CoRR(2024)
Abstract
We introduce SynGround, a novel framework that combines data-driven learning
and knowledge transfer from various large-scale pretrained models to enhance
the visual grounding capabilities of a pretrained vision-and-language model.
The knowledge transfer from the models initiates the generation of image
descriptions through an image description generator. These descriptions serve
dual purposes: they act as prompts for synthesizing images through a
text-to-image generator, and as queries for synthesizing text, from which
phrases are extracted using a large language model. Finally, we leverage an
open-vocabulary object detector to generate synthetic bounding boxes for the
synthetic images and texts. We finetune a pretrained vision-and-language model
on this dataset by optimizing a mask-attention consistency objective that
aligns region annotations with gradient-based model explanations. The resulting
model improves the grounding capabilities of an off-the-shelf
vision-and-language model. Particularly, SynGround improves the pointing game
accuracy of ALBEF on the Flickr30k dataset from 79.38
RefCOCO+ Test A from 69.35
63.67
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined