CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models

The advancement of large language models (LLMs) has enhanced the ability togeneralize across a wide range of unseen natural language processing (NLP)tasks through instruction-following. Yet, their effectiveness often diminishesin low-resource languages like Chinese, exacerbated by biased evaluations fromdata leakage, casting doubt on their true generalizability to new linguisticterritories. In response, we introduce the Chinese Instruction-FollowingBenchmark (CIF-Bench), designed to evaluate the zero-shot generalizability ofLLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000input-output pairs, developed by native speakers to test complex reasoning andChinese cultural nuances across 20 categories. To mitigate data contamination,we release only half of the dataset publicly, with the remainder kept private,and introduce diversified instructions to minimize score variance, totaling45,000 data instances. Our evaluation of 28 selected LLMs reveals a noticeableperformance gap, with the best model scoring only 52.9limitations of LLMs in less familiar language and task contexts. This work notonly uncovers the current limitations of LLMs in handling Chinese languagetasks but also sets a new standard for future LLM generalizability research,pushing towards the development of more adaptable, culturally informed, andlinguistically diverse models.