CHAI: Clustered Head Attention for Efficient LLM Inference
arxiv(2024)
摘要
Large Language Models (LLMs) with hundreds of billions of parameters have
transformed the field of machine learning. However, serving these models at
inference time is both compute and memory intensive, where a single request can
require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is
one of the key components of LLMs, which can account for over 50
memory and compute requirement. We observe that there is a high amount of
redundancy across heads on which tokens they pay attention to. Based on this
insight, we propose Clustered Head Attention (CHAI). CHAI combines heads with a
high amount of correlation for self-attention at runtime, thus reducing both
memory and compute. In our experiments, we show that CHAI is able to reduce the
memory requirements for storing K,V cache by up to 21.4
latency by up to 1.73x without any fine-tuning required. CHAI achieves this
with a maximum 3.2
OPT-66B, LLAMA-7B, LLAMA-33B) and 5 different evaluation datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要