Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1 Long Papers)(2024)
Abstract
We study semi-supervised sequence generation tasks, where the few labeledexamples are too scarce to finetune a model, and meanwhile, few-shot promptedlarge language models (LLMs) exhibit room for improvement. In this paper, wepresent the discovery that a student model distilled from a few-shot promptedLLM can commonly generalize better than its teacher to unseen examples on suchtasks. We find that the student is able to learn a general pattern from thehigh-quality pseudolabels produced by the teacher during knowledge distillation(KD), and favorably not a general pattern from the low-quality pseudolables.Leveraging this discovery, we propose a new method, Multistage CollaborativeKnowledge Distillation from an LLM (MCKD), for these tasks. MCKD first few-shotprompts an LLM to produce pseudolabels for unlabeled data. Then at each stageof an iterative KD process, a new pair of students is trained on disjointpartitions of the pseudolabeled data, and produces new and improvedpseudolabels for their unseen partitions. We conduct extensive experiments onfour syntactic and semantic parsing datasets and show the effectiveness of MCKDfor low-resource semi-supervised sequence generation. On CRAFT biomedicalparsing, for example, 3-stage MCKD with 50 labeled examples outperforms an LLMteacher and vanilla KD by 7.5the performance of supervised finetuning with 500 labeled examples.
MoreTranslated text
PDF
View via Publisher
AI Read Science
Must-Reading Tree
Example

Generate MRT to find the research sequence of this paper
Data Disclaimer
The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn
Chat Paper
去 AI 文献库 对话