Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
CoRR(2024)
摘要
Compared to Full-Model Fine-Tuning (FMFT), Parameter Efficient Fine-Tuning
(PEFT) has demonstrated superior performance and lower computational overhead
in several code understanding tasks, such as code summarization and code
search. This advantage can be attributed to PEFT's ability to alleviate the
catastrophic forgetting issue of Pre-trained Language Models (PLMs) by updating
only a small number of parameters. As a result, PEFT effectively harnesses the
pre-trained general-purpose knowledge for downstream tasks. However, existing
studies primarily involve static code comprehension, aligning with the
pre-training paradigm of recent PLMs and facilitating knowledge transfer, but
they do not account for dynamic code changes. Thus, it remains unclear whether
PEFT outperforms FMFT in task-specific adaptation for code-change-related
tasks. To address this question, we examine two prevalent PEFT methods, namely
Adapter Tuning (AT) and Low-Rank Adaptation (LoRA), and compare their
performance with FMFT on five popular PLMs. Specifically, we evaluate their
performance on two widely-studied code-change-related tasks: Just-In-Time
Defect Prediction (JIT-DP) and Commit Message Generation (CMG). The results
demonstrate that both AT and LoRA achieve state-of-the-art (SOTA) results in
JIT-DP and exhibit comparable performances in CMG when compared to FMFT and
other SOTA approaches. Furthermore, AT and LoRA exhibit superiority in
cross-lingual and low-resource scenarios. We also conduct three probing tasks
to explain the efficacy of PEFT techniques on JIT-DP and CMG tasks from both
static and dynamic perspectives. The study indicates that PEFT, particularly
through the use of AT and LoRA, offers promising advantages in
code-change-related tasks, surpassing FMFT in certain aspects.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要