Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency
arxiv(2024)
摘要
Collecting labeled datasets in finance is challenging due to scarcity of
domain experts and higher cost of employing them. While Large Language Models
(LLMs) have demonstrated remarkable performance in data annotation tasks on
general domain datasets, their effectiveness on domain specific datasets
remains underexplored. To address this gap, we investigate the potential of
LLMs as efficient data annotators for extracting relations in financial
documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2,
and MPT Instruct) against expert annotators and crowdworkers. We demonstrate
that the current state-of-the-art LLMs can be sufficient alternatives to
non-expert crowdworkers. We analyze models using various prompts and parameter
settings and find that customizing the prompts for each relation group by
providing specific examples belonging to those groups is paramount.
Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify
outputs that may require expert attention. Finally, we perform an extensive
time, cost and error analysis and provide recommendations for the collection
and usage of automated annotations in domain-specific settings.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要