Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
CoRR(2024)
摘要
Large Language Models (LLMs) are often described as being instances of
foundation models - that is, models that transfer strongly across various tasks
and conditions in few-show or zero-shot manner, while exhibiting scaling laws
that predict function improvement when increasing the pre-training scale. These
claims of excelling in different functions and tasks rely on measurements taken
across various sets of standardized benchmarks showing high scores for such
models. We demonstrate here a dramatic breakdown of function and reasoning
capabilities of state-of-the-art models trained at the largest available scales
which claim strong function, using a simple, short, conventional common sense
problem formulated in concise natural language, easily solvable by humans. The
breakdown is dramatic, as models also express strong overconfidence in their
wrong solutions, while providing often non-sensical "reasoning"-like
explanations akin to confabulations to justify and backup the validity of their
clearly failed responses, making them sound plausible. Various standard
interventions in an attempt to get the right solution, like various type of
enhanced prompting, or urging the models to reconsider the wrong solutions
again by multi step re-evaluation, fail. We take these initial observations to
the scientific and technological community to stimulate urgent re-assessment of
the claimed capabilities of current generation of LLMs, Such re-assessment also
requires common action to create standardized benchmarks that would allow
proper detection of such basic reasoning deficits that obviously manage to
remain undiscovered by current state-of-the-art evaluation procedures and
benchmarks. Code for reproducing experiments in the paper and raw experiments
data can be found at https://github.com/LAION-AI/AIW
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要