Improving system latency of AI accelerator with on-chip pipelined activation preprocessing and multi-mode batch inference

State-of-the-art neural network accelerators exploit massive computing parallelism to achieve high throughput. However, significant latency is observed on the master-slave-based AI acceleration system which limits its adaptation in real-time applications. Investigation in de-facto GPU system reveals tremendous timing overhead for preprocessing of input activations, which is commonly executed on th...