Axioms for AI Alignment from Human Feedback
CoRR(2024)
Abstract
In the context of reinforcement learning from human feedback (RLHF), the
reward function is generally derived from maximum likelihood estimation of a
random utility model based on pairwise comparisons made by humans. The problem
of learning a reward function is one of preference aggregation that, we argue,
largely falls within the scope of social choice theory. From this perspective,
we can evaluate different aggregation methods via established axioms, examining
whether these methods meet or fail well-known standards. We demonstrate that
both the Bradley-Terry-Luce Model and its broad generalizations fail to meet
basic axioms. In response, we develop novel rules for learning reward functions
with strong axiomatic guarantees. A key innovation from the standpoint of
social choice is that our problem has a linear structure, which greatly
restricts the space of feasible rules and leads to a new paradigm that we call
linear social choice.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined