WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models
arxiv(2024)
摘要
The factuality of large language model (LLMs) tends to decay over time since
events posterior to their training are "unknown" to them. One way to keep
models up-to-date could be factual update: the task of inserting, replacing, or
removing certain simple (atomic) facts within the model. To study this task, we
present WikiFactDiff, a dataset that describes the evolution of factual
knowledge between two dates as a collection of simple facts divided into three
categories: new, obsolete, and static. We describe several update scenarios
arising from various combinations of these three types of basic update. The
facts are represented by subject-relation-object triples; indeed, WikiFactDiff
was constructed by comparing the state of the Wikidata knowledge base at 4
January 2021 and 27 February 2023. Those fact are accompanied by verbalization
templates and cloze tests that enable running update algorithms and their
evaluation metrics. Contrary to other datasets, such as zsRE and CounterFact,
WikiFactDiff constitutes a realistic update setting that involves various
update scenarios, including replacements, archival, and new entity insertions.
We also present an evaluation of existing update algorithms on WikiFactDiff.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要