GSVA: Generalized Segmentation via Multimodal Large Language Models
CoRR(2023)
摘要
Generalized Referring Expression Segmentation (GRES) extends the scope of
classic RES to referring to multiple objects in one expression or identifying
the empty targets absent in the image. GRES poses challenges in modeling the
complex spatial relationships of the instances in the image and identifying
non-existing referents. Recently, Multimodal Large Language Models (MLLMs) have
shown tremendous progress in these complicated vision-language tasks.
Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient
in understanding contexts with visual inputs. Among them, LISA, as a
representative, adopts a special [SEG] token to prompt a segmentation mask
decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing
solutions to of GRES remain unsatisfactory since current segmentation MLLMs
cannot properly handle the cases where users might reference multiple subjects
in a singular prompt or provide descriptions incongruent with any image target.
In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to
address this gap. Specifically, GSVA reuses the [SEG] token to prompt the
segmentation model towards supporting multiple mask references simultaneously
and innovatively learns to generate a [REJ] token to reject the null targets
explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue,
marking a notable enhancement and setting a new record on the GRES benchmark
gRefCOCO dataset. GSVA also proves effective across various classic referring
expression segmentation and comprehension tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要