KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance Domains
CoRR(2023)
摘要
We introduce KnowledgeMath, a novel benchmark designed to evaluate LLMs'
capabilities in applying financial knowledge to solve complex math word
problems. Compared to prior works, this study features three core advancements.
First, KnowledgeMath includes 1,259 problems with a hybrid of textual and
tabular content and require college-level knowledge in the finance domain for
effective resolution. Second, we provide expert-annotated, detailed solution
references in Python program format, ensuring a high-quality benchmark for LLM
assessment. Finally, we evaluate a wide spectrum of 14 LLMs with different
prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. The
current best-performing system (i.e., GPT-4 with Program-of-Thoughts) achieves
only 45.4% accuracy, leaving substantial room for improvement. While
knowledge-augmented LLMs can improve the performance (e.g., from 23.9% to 32.0%
for GPT-3.5), it is still significantly lower the estimated human expert
performance of 94%. We believe that KnowledgeMath can facilitate future
research on domain-specific knowledge retrieval and augmentation into the math
word problem-solving process. We will release the benchmark and code at
https://github.com/yale-nlp/KnowledgeMath.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要