MuLan: A Study of Fact Mutability in Language Models
arxiv(2024)
摘要
Facts are subject to contingencies and can be true or false in different
circumstances. One such contingency is time, wherein some facts mutate over a
given period, e.g., the president of a country or the winner of a championship.
Trustworthy language models ideally identify mutable facts as such and process
them accordingly. We create MuLan, a benchmark for evaluating the ability of
English language models to anticipate time-contingency, covering both 1:1 and
1:N relations. We hypothesize that mutable facts are encoded differently than
immutable ones, hence being easier to update. In a detailed evaluation of six
popular large language models, we consistently find differences in the LLMs'
confidence, representations, and update behavior, depending on the mutability
of a fact. Our findings should inform future work on the injection of and
induction of time-contingent knowledge to/from LLMs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要