Holmes: Benchmark the Linguistic Competence of Language Models
arxiv(2024)
摘要
We introduce Holmes, a benchmark to assess the linguistic competence of
language models (LMs) - their ability to grasp linguistic phenomena. Unlike
prior prompting-based evaluations, Holmes assesses the linguistic competence of
LMs via their internal representations using classifier-based probing. In doing
so, we disentangle specific phenomena (e.g., part-of-speech of words) from
other cognitive abilities, like following textual instructions, and meet recent
calls to assess LMs' linguistic competence in isolation. Composing Holmes, we
review over 250 probing studies and feature more than 200 datasets to assess
syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing
over 50 LMs reveals that, aligned with known trends, their linguistic
competence correlates with model size. However, surprisingly, model
architecture and instruction tuning also significantly influence performance,
particularly in morphology and syntax. Finally, we propose FlashHolmes, a
streamlined version of Holmes designed to lower the high computation load while
maintaining high-ranking precision.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要