Prompt Perturbation Consistency Learning for Robust Language Models
Conference of the European Chapter of the Association for Computational Linguistics(2024)
摘要
Large language models (LLMs) have demonstrated impressive performance on a
number of natural language processing tasks, such as question answering and
text summarization. However, their performance on sequence labeling tasks such
as intent classification and slot filling (IC-SF), which is a central component
in personal assistant systems, lags significantly behind discriminative models.
Furthermore, there is a lack of substantive research on the robustness of LLMs
to various perturbations in the input prompts. The contributions of this paper
are three-fold. First, we show that fine-tuning sufficiently large LLMs can
produce IC-SF performance comparable to discriminative models. Next, we
systematically analyze the performance deterioration of those fine-tuned models
due to three distinct yet relevant types of input perturbations - oronyms,
synonyms, and paraphrasing. Finally, we propose an efficient mitigation
approach, Prompt Perturbation Consistency Learning (PPCL), which works by
regularizing the divergence between losses from clean and perturbed samples.
Our experiments demonstrate that PPCL can recover on average 59
performance drop for IC and SF tasks, respectively. Furthermore, PPCL beats the
data augmentation approach while using ten times fewer augmented data samples.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要