LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
CoRR(2024)
摘要
Automatic side-by-side evaluation has emerged as a promising approach to
evaluating the quality of responses from large language models (LLMs). However,
analyzing the results from this evaluation approach raises scalability and
interpretability challenges. In this paper, we present LLM Comparator, a novel
visual analytics tool for interactively analyzing results from automatic
side-by-side evaluation. The tool supports interactive workflows for users to
understand when and why a model performs better or worse than a baseline model,
and how the responses from two models are qualitatively different. We
iteratively designed and developed the tool by closely working with researchers
and engineers at a large technology company. This paper details the user
challenges we identified, the design and development of the tool, and an
observational study with participants who regularly evaluate their models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要