AIMMI: Audio and Image Multi-Modal Intelligence via a Low-Power SoC With 2-MByte On-Chip MRAM for IoT Devices

In this article, we present an ultra-low-power multi-modal signal processing system on chip (SoC) audio and image multi-modal intelligence (AIMMI) that integrates a versatile deep neural network (DNN) engine with audio and image signal processing accelerators for multi-modal Internet-of-Things (IoT) intelligence. In order to get high energy efficiency under resource-constrained IoT scenarios, AIMMI features three efficiency-boosting techniques: 1) 2-MB on-chip non-volatile magnetoresistive RAM (MRAM) to store all DNN weights with MRAM-cache microarchitecture that incorporates dynamic power gating to reduce both leakage and dynamic power consumption; 2) a deliberate power management scheme that enables optimized power modes under different operating situations; and 3) a novel reconfigurable neural engine (NE) with energy-efficient dataflow for comprehensive DNN instructions. Fabricated in TSMC 22-nm ultra-low leakage (ULL) technology with MRAM, AIMMI achieves up to 3–10-TOPS/W peak energy efficiency and consumes only 0.25–3.84 mW. It demonstrates convolutional neural network (CNN), generative adversarial network (GAN), and back-propagation (BP) operations on a single accelerator SoC for multi-modal fusion, outperforming state-of-the-art DNN processors by 1.4 $\times$ –4.5 $\times$ in energy efficiency.