Identifying Factual Inconsistency in Summaries: Towards Effective Utilization of Large Language Model
CoRR(2024)
摘要
Factual inconsistency poses a significant hurdle for the commercial
deployment of abstractive summarizers. Under this Large Language Model (LLM)
era, this work focuses around two important questions: what is the best way to
leverage LLM for factual inconsistency detection, and how could we distill a
smaller LLM with both high efficiency and efficacy? Three zero-shot paradigms
are firstly proposed and evaluated across five diverse datasets: direct
inference on the entire summary or each summary window; entity verification
through question generation and answering. Experiments suggest that LLM itself
is capable to resolve this task train-free under the proper paradigm design,
surpassing strong trained baselines by 2.8
practical utility, we then propose training strategies aimed at distilling
smaller open-source LLM that learns to score the entire summary at once with
high accuracy, which outperforms the zero-shot approaches by much larger LLM,
serving as an effective and efficient ready-to-use scorer.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要