Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
CoRR(2024)
摘要
Post-Training Quantization (PTQ) enhances the efficiency of Large Language
Models (LLMs) by enabling faster operation and compatibility with more
accessible hardware through reduced memory usage, at the cost of small
performance drops. We explore the role of calibration sets in PTQ, specifically
their effect on hidden activations in various notable open-source LLMs.
Calibration sets are crucial for evaluating activation magnitudes and
identifying outliers, which can distort the quantization range and negatively
impact performance. Our analysis reveals a marked contrast in quantization
effectiveness across models. The older OPT model, which much of the
quantization literature is based on, shows significant performance
deterioration and high susceptibility to outliers with varying calibration
sets. In contrast, newer models like Llama-2 7B, Llama-3 8B, Command-R 35B, and
Mistral 7B demonstrate strong robustness, with Mistral 7B showing near-immunity
to outliers and stable activations. These findings suggest a shift in PTQ
strategies might be needed. As advancements in pre-training methods reduce the
relevance of outliers, there is an emerging need to reassess the fundamentals
of current quantization literature. The emphasis should pivot towards
optimizing inference speed, rather than primarily focusing on outlier
preservation, to align with the evolving characteristics of state-of-the-art
LLMs.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要