ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer
CoRR(2024)
摘要
Stochastic computing (SC) has emerged as a promising computing paradigm for
neural acceleration. However, how to accelerate the state-of-the-art Vision
Transformer (ViT) with SC remains unclear. Unlike convolutional neural
networks, ViTs introduce notable compatibility and efficiency challenges
because of their nonlinear functions, e.g., softmax and Gaussian Error Linear
Units (GELU). In this paper, for the first time, a ViT accelerator based on
end-to-end SC, dubbed ASCEND, is proposed. ASCEND co-designs the SC circuits
and ViT networks to enable accurate yet efficient acceleration. To overcome the
compatibility challenges, ASCEND proposes a novel deterministic SC block for
GELU and leverages an SC-friendly iterative approximate algorithm to design an
accurate and efficient softmax circuit. To improve inference efficiency, ASCEND
develops a two-stage training pipeline to produce accurate low-precision ViTs.
With extensive experiments, we show the proposed GELU and softmax blocks
achieve 56.3
respectively and reduce the area-delay product (ADP) by 5.29x and 12.6x,
respectively. Moreover, compared to the baseline low-precision ViTs, ASCEND
also achieves significant accuracy improvements on CIFAR10 and CIFAR100.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要