On Inter-dataset Code Duplication and Data Leakage in Large Language Models
CoRR(2024)
摘要
Motivation. Large language models (LLMs) have exhibited remarkable
proficiency in diverse software engineering (SE) tasks. Handling such tasks
typically involves acquiring foundational coding knowledge on large,
general-purpose datasets during a pre-training phase, and subsequently refining
on smaller, task-specific datasets as part of a fine-tuning phase.
Problem statement. Data leakage is a well-known issue in training of machine
learning models. A manifestation of this issue is the intersection of the
training and testing splits. While intra-dataset code duplication examines this
intersection within a given dataset and has been addressed in prior research,
inter-dataset code duplication, which gauges the overlap between different
datasets, remains largely unexplored. If this phenomenon exists, it could
compromise the integrity of LLM evaluations because of the inclusion of
fine-tuning test samples that were already encountered during pre-training,
resulting in inflated performance metrics.
Contribution. This paper explores the phenomenon of inter-dataset code
duplication and its impact on evaluating LLMs across diverse SE tasks.
Study design. We conduct an empirical study using the CSN dataset, a widely
adopted pre-training dataset, and five fine-tuning datasets used for various SE
tasks. We first identify the intersection between the pre-training and
fine-tuning datasets using a deduplication process. Then, we fine-tune four
models pre-trained on CSN to evaluate their performance on samples encountered
during pre-training and those unseen during that phase.
Results. Our findings reveal a potential threat to the evaluation of various
LLMs across multiple SE tasks, stemming from the inter-dataset code duplication
phenomenon. Moreover, we demonstrate that this threat is accentuated by factors
like the LLM's size and the chosen fine-tuning technique.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要