Evaluating Language Model Agency through Negotiations
CoRR(2024)
摘要
Companies, organizations, and governments increasingly exploit Language
Models' (LM) remarkable capability to display agent-like behavior. As LMs are
adopted to perform tasks with growing autonomy, there exists an urgent need for
reliable and scalable evaluation benchmarks. Current, predominantly static LM
benchmarks are ill-suited to evaluate such dynamic applications. Thus, we
propose jointly evaluating LM performance and alignment through the lenses of
negotiation games. We argue that this common task better reflects real-world
deployment conditions while offering insights into LMs' decision-making
processes. Crucially, negotiation games allow us to study multi-turn, and
cross-model interactions, modulate complexity, and side-step accidental data
leakage in evaluation. We report results for six publicly accessible LMs from
several major providers on a variety of negotiation games, evaluating both
self-play and cross-play performance. Noteworthy findings include: (i)
open-source models are currently unable to complete these tasks; (ii)
cooperative bargaining games prove challenging; and (iii) the most powerful
models do not always "win".
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要