Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding
IJCAI 2023(2023)
摘要
Panoptic narrative grounding (PNG) aims to segment things and stuff objects
in an image described by noun phrases of a narrative caption. As a multimodal
task, an essential aspect of PNG is the visual-linguistic interaction between
image and caption. The previous two-stage method aggregates visual contexts
from offline-generated mask proposals to phrase features, which tend to be
noisy and fragmentary. The recent one-stage method aggregates only pixel
contexts from image features to phrase features, which may incur semantic
misalignment due to lacking object priors. To realize more comprehensive
visual-linguistic interaction, we propose to enrich phrases with coupled pixel
and object contexts by designing a Phrase-Pixel-Object Transformer Decoder
(PPO-TD), where both fine-grained part details and coarse-grained entity clues
are aggregated to phrase features. In addition, we also propose a PhraseObject
Contrastive Loss (POCL) to pull closer the matched phrase-object pairs and push
away unmatched ones for aggregating more precise object contexts from more
phrase-relevant object tokens. Extensive experiments on the PNG benchmark show
our method achieves new state-of-the-art performance with large margins.
更多查看译文
关键词
panoptic
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要