Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark
CoRR(2024)
摘要
The summarization capabilities of pretrained and large language models (LLMs)
have been widely validated in general areas, but their use in scientific
corpus, which involves complex sentences and specialized knowledge, has been
less assessed. This paper presents conceptual and experimental analyses of
scientific summarization, highlighting the inadequacies of traditional
evaluation methods, such as n-gram, embedding comparison, and QA,
particularly in providing explanations, grasping scientific concepts, or
identifying key content. Subsequently, we introduce the Facet-aware Metric
(FM), employing LLMs for advanced semantic matching to evaluate summaries based
on different aspects. This facet-aware approach offers a thorough evaluation of
abstracts by decomposing the evaluation task into simpler subtasks.Recognizing
the absence of an evaluation benchmark in this domain, we curate a Facet-based
scientific summarization Dataset (FD) with facet-level annotations. Our
findings confirm that FM offers a more logical approach to evaluating
scientific summaries. In addition, fine-tuned smaller models can compete with
LLMs in scientific contexts, while LLMs have limitations in learning from
in-context information in scientific domains. This suggests an area for future
enhancement of LLMs.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要