System 2 Attention (is something you might need too).
CoRR(2023)
摘要
Soft attention in Transformer-based Large Language Models (LLMs) is
susceptible to incorporating irrelevant information from the context into its
latent representations, which adversely affects next token generations. To help
rectify these issues, we introduce System 2 Attention (S2A), which leverages
the ability of LLMs to reason in natural language and follow instructions in
order to decide what to attend to. S2A regenerates the input context to only
include the relevant portions, before attending to the regenerated context to
elicit the final response. In experiments, S2A outperforms standard
attention-based LLMs on three tasks containing opinion or irrelevant
information, QA, math word problems and longform generation, where S2A
increases factuality and objectivity, and decreases sycophancy.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要