AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
CoRR(2024)
摘要
Humans regularly engage in analogical thinking, relating personal experiences
to current situations (X is analogous to Y because of Z). Analogical
thinking allows humans to solve problems in creative ways, grasp difficult
concepts, and articulate ideas more effectively. Can language models (LMs) do
the same? To answer this question, we propose ANALOBENCH, a benchmark to
determine analogical reasoning ability in LMs. Our benchmarking approach
focuses on aspects of this ability that are common among humans: (i) recalling
related experiences from a large amount of information, and (ii) applying
analogical reasoning to complex and lengthy scenarios. We test a broad
collection of proprietary models (e.g., GPT family, Claude V2) and open source
models such as LLaMA2. As in prior results, scaling up LMs results in some
performance boosts. Surprisingly, scale offers minimal gains when, (i)
analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from
a large pool of information, a process analogous to finding a needle in a
haystack. We hope these observations encourage further research in this field.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要