Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs
CoRR(2024)
摘要
The attribution of question answering is to provide citations for supporting
generated statements, and has attracted wide research attention. The current
methods for automatically evaluating the attribution, which are often based on
Large Language Models (LLMs), are still inadequate, particularly in recognizing
subtle differences between attributions, and complex relationships between
citations and statements. To compare these attribution evaluation methods and
develop new ones, we introduce a set of fine-grained categories (i.e.,
supportive, insufficient, contradictory and irrelevant) for measuring the
attribution, and develop a Complex Attributed Question Answering (CAQA)
benchmark by leveraging knowledge graphs (KGs) for automatically generating
attributions of different categories to question-answer pairs. Our analysis
reveals that existing evaluators perform poorly under fine-grained attribution
settings and exhibit weaknesses in complex citation-statement reasoning. Our
CAQA benchmark, validated with human annotations, emerges as a promising tool
for selecting and developing LLM attribution evaluators.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要