Retrieval is Accurate Generation
ICLR 2024(2024)
摘要
Standard language models generate text by selecting tokens from a fixed,
finite, and standalone vocabulary. We introduce a novel method that selects
context-aware phrases from a collection of supporting documents. One of the
most significant challenges for this paradigm shift is determining the training
oracles, because a string of text can be segmented in various ways and each
segment can be retrieved from numerous possible documents. To address this, we
propose to initialize the training oracles using linguistic heuristics and,
more importantly, bootstrap the oracles through iterative self-reinforcement.
Extensive experiments show that our model not only outperforms standard
language models on a variety of knowledge-intensive tasks but also demonstrates
improved generation quality in open-ended text generation. For instance,
compared to the standard language model counterpart, our model raises the
accuracy from 23.47
42.61
achieves the best performance and the lowest latency among several
retrieval-augmented baselines. In conclusion, we assert that retrieval is more
accurate generation and hope that our work will encourage further research on
this new paradigm shift.
更多查看译文
关键词
Artificial Intelligence,Natural Language Processing,Text Generation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要