Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
arxiv(2024)
摘要
When training deep neural networks, the phenomenon of dying
neurons x2013units that become inactive or saturated, output zero
during trainingx2013 has traditionally been viewed as undesirable,
linked with optimization challenges, and contributing to plasticity loss in
continual learning scenarios. In this paper, we reassess this phenomenon,
focusing on sparsity and pruning. By systematically exploring the impact of
various hyperparameter configurations on dying neurons, we unveil their
potential to facilitate simple yet effective structured pruning algorithms. We
introduce Demon Pruning (DemP), a method that controls the
proliferation of dead neurons, dynamically leading to network sparsity.
Achieved through a combination of noise injection on active units and a
one-cycled schedule regularization strategy, DemP stands out for its simplicity
and broad applicability. Experiments on CIFAR10 and ImageNet datasets
demonstrate that DemP surpasses existing structured pruning techniques,
showcasing superior accuracy-sparsity tradeoffs and training speedups. These
findings suggest a novel perspective on dying neurons as a valuable resource
for efficient model compression and optimization.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要