Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences
arxiv(2022)
摘要
Lack of factual correctness is an issue that still plagues state-of-the-art
summarization systems despite their impressive progress on generating seemingly
fluent summaries. In this paper, we show that factual inconsistency can be
caused by irrelevant parts of the input text, which act as confounders. To that
end, we leverage information-theoretic measures of causal effects to quantify
the amount of confounding and precisely quantify how they affect the
summarization performance. Based on insights derived from our theoretical
results, we design a simple multi-task model to control such confounding by
leveraging human-annotated relevant sentences when available. Crucially, we
give a principled characterization of data distributions where such confounding
can be large thereby necessitating the use of human annotated relevant
sentences to generate factual summaries. Our approach improves faithfulness
scores by 20% over strong baselines on AnswerSumm
, a conversation summarization dataset where lack
of faithfulness is a significant issue due to the subjective nature of the
task. Our best method achieves the highest faithfulness score while also
achieving state-of-the-art results on standard metrics like ROUGE and METEOR.
We corroborate these improvements through human evaluation.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要