Measuring Vision-Language STEM Skills of Neural Models
ICLR 2024(2024)
摘要
We introduce a new challenge to test the STEM skills of neural models. The
problems in the real world often require solutions, combining knowledge from
STEM (science, technology, engineering, and math). Unlike existing datasets,
our dataset requires the understanding of multimodal vision-language
information of STEM. Our dataset features one of the largest and most
comprehensive datasets for the challenge. It includes 448 skills and 1,073,146
questions spanning all STEM subjects. Compared to existing datasets that often
focus on examining expert-level ability, our dataset includes fundamental
skills and questions designed based on the K-12 curriculum. We also add
state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our
benchmark. Results show that the recent model advances only help master a very
limited number of lower grade-level skills (2.5
dataset. In fact, these models are still well below (averaging 54.7
performance of elementary students, not to mention near expert-level
performance. To understand and increase the performance on our dataset, we
teach the models on a training split of our dataset. Even though we observe
improved performance, the model performance remains relatively low compared to
average elementary students. To solve STEM problems, we will need novel
algorithmic innovations from the community.
更多查看译文
关键词
Benchmark,STEM,Multimodal,Vision-language models,Language models
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要