GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network
CoRR(2024)
摘要
Large Language Models (LLMs) exhibit strong In-Context Learning (ICL)
capabilities when prompts with demonstrations are applied to them. However,
fine-tuning still remains crucial to further enhance their adaptability.
Prompt-based fine-tuning proves to be an effective fine-tuning method in
low-data scenarios, but high demands on computing resources limit its
practicality. We address this issue by introducing a prompt-based
parameter-efficient fine-tuning (PEFT) approach. GNNavi leverages insights into
ICL's information flow dynamics, which indicates that label words act in
prompts as anchors for information propagation. GNNavi employs a Graph Neural
Network (GNN) layer to precisely guide the aggregation and distribution of
information flow during the processing of prompts by hardwiring the desired
information flow into the GNN. Our experiments on text classification tasks
with GPT-2 and Llama2 shows GNNavi surpasses standard prompt-based fine-tuning
methods in few-shot settings by updating just 0.2
compare GNNavi with prevalent PEFT approaches, such as prefix tuning, LoRA and
Adapter in terms of performance and efficiency. Our analysis reveals that
GNNavi enhances information flow and ensures a clear aggregation process.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要