Dual-View Visual Contextualization for Web Navigation
CoRR(2024)
摘要
Automatic web navigation aims to build a web agent that can follow language
instructions to execute complex and diverse tasks on real-world websites.
Existing work primarily takes HTML documents as input, which define the
contents and action spaces (i.e., actionable elements and operations) of
webpages. Nevertheless, HTML documents may not provide a clear task-related
context for each element, making it hard to select the right (sequence of)
actions. In this paper, we propose to contextualize HTML elements through their
"dual views" in webpage screenshots: each HTML element has its corresponding
bounding box and visual content in the screenshot. We build upon the insight –
web developers tend to arrange task-related elements nearby on webpages to
enhance user experiences – and propose to contextualize each element with its
neighbor elements, using both textual and visual features. The resulting
representations of HTML elements are more informative for the agent to take
action. We validate our method on the recently released Mind2Web dataset, which
features diverse navigation domains and tasks on real-world websites. Our
method consistently outperforms the baseline in all the scenarios, including
cross-task, cross-website, and cross-domain ones.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要