Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
CoRR(2024)
摘要
Existing question answering (QA) datasets are no longer challenging to most
powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA,
NaturalQuestions, ELI5 and HotpotQA mainly study “known unknowns” with clear
indications of both what information is missing, and how to find it to answer
the question. Hence, good performance on these benchmarks provides a false
sense of security. A yet unmet need of the NLP community is a bank of
non-factoid, multi-perspective questions involving a great deal of unclear
information needs, i.e. “unknown uknowns”. We claim we can find such
questions in search engine logs, which is surprising because most
question-intent queries are indeed factoid. We present Researchy Questions, a
dataset of search engine queries tediously filtered to be non-factoid,
“decompositional” and multi-perspective. We show that users spend a lot of
“effort” on these questions in terms of signals like clicks and session
length, and that they are also challenging for GPT-4. We also show that “slow
thinking” answering techniques, like decomposition into sub-questions shows
benefit over answering directly. We release ∼ 100k Researchy Questions,
along with the Clueweb22 URLs that were clicked.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要