KTO: Model Alignment as Prospect Theoretic Optimization
CoRR(2024)
摘要
Kahneman Tversky's prospect theory tells us that humans perceive
random variables in a biased but well-defined manner; for example, humans are
famously loss-averse. We show that objectives for aligning LLMs with human
feedback implicitly incorporate many of these biases – the success of these
objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed
to them being human-aware loss functions (HALOs). However, the
utility functions these methods attribute to humans still differ from those in
the prospect theory literature. Using a Kahneman-Tversky model of human
utility, we propose a HALO that directly maximizes the utility of generations
instead of maximizing the log-likelihood of preferences, as current methods do.
We call this approach Kahneman-Tversky Optimization (KTO), and it matches or
exceeds the performance of preference-based methods at scales from 1B to 30B.
Crucially, KTO does not need preferences – only a binary signal of whether an
output is desirable or undesirable for a given input. This makes it far easier
to use in the real world, where preference data is scarce and expensive.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要