ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
arxiv(2023)
摘要
While existing large vision-language multimodal models focus on whole image
understanding, there is a prominent gap in achieving region-specific
comprehension. Current approaches that use textual coordinates or spatial
encodings often fail to provide a user-friendly interface for visual prompting.
To address this challenge, we introduce a novel multimodal model capable of
decoding arbitrary visual prompts. This allows users to intuitively mark images
and interact with the model using natural cues like a "red bounding box" or
"pointed arrow". Our simple design directly overlays visual markers onto the
RGB image, eliminating the need for complex region encodings, yet achieves
state-of-the-art performance on region-understanding tasks like Visual7W,
PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present
ViP-Bench, a comprehensive benchmark to assess the capability of models in
understanding visual prompts across multiple dimensions, enabling future
research in this domain. Code, data, and model are publicly available.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要