IterAlign: Iterative Constitutional Alignment of Large Language Models
arxiv(2024)
摘要
With the rapid development of large language models (LLMs), aligning LLMs
with human values and societal norms to ensure their reliability and safety has
become crucial. Reinforcement learning with human feedback (RLHF) and
Constitutional AI (CAI) have been proposed for LLM alignment. However, these
methods require either heavy human annotations or explicitly pre-defined
constitutions, which are labor-intensive and resource-consuming. To overcome
these drawbacks, we study constitution-based LLM alignment and propose a
data-driven constitution discovery and self-alignment framework called
IterAlign. IterAlign leverages red teaming to unveil the weaknesses of an LLM
and automatically discovers new constitutions using a stronger LLM. These
constitutions are then used to guide self-correction of the base LLM. Such a
constitution discovery pipeline can be run iteratively and automatically to
discover new constitutions that specifically target the alignment gaps in the
current LLM. Empirical results on several safety benchmark datasets and
multiple base LLMs show that IterAlign successfully improves truthfulness,
helpfulness, harmlessness and honesty, improving the LLM alignment by up to
13.5% in harmlessness.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要