A Heterogeneous RRAM In-Memory and SRAM Near-Memory SoC for Fused Frame and Event-Based Target Identification and Tracking

Accurate identification of the target and tracking it at high speeds using drone-mounted cameras and compute hardware finds military and commercial applications. Conventional frame-based cameras and convolutional neural networks (CNNs) extract detailed spatial information to show high accuracy but suffer from lower throughput caused by large models. Alternatively, event cameras capture the motion information as an asynchronous event stream with high temporal resolution. Spiking neural networks (SNNs) can be used to process these data at high speed, but the sparse sensing and difficulty in training SNN limit the accuracy. Fusing the complementary spatial and temporal advantages of the frame and event-based pipelines allows high-speed identification and tracking while preserving accuracy. The SNN processes the event stream continuously to provide high-speed target estimates with lower accuracy, while periodic anchors provided by the reliable CNN restore the accuracy. In this work, we present a heterogeneous programmable ARM Cortex-based system-on-a-chip (SoC) in 40-nm Taiwan Semiconductor Manufacturing Company (TSMC) ultra low power (ULP) technology with power-efficient RRAM compute-in-memory (CIM) for CNN and high-speed SRAM compute-near-memory (CNM) for SNN for the modality-matched acceleration of the hybrid vision. Our SoC incorporates: 1) two levels of power gating to save 91.8% of total chip power with non-volatile RRAM-CIM; 2) embedded triple error correction (TEC) within RRAM CIM macro to suppress the raw bit errors in reading by > 5 orders of magnitude; and 3) parallelly operating CNN and SNN modules to provide > 100 outputs/s. Such cross-layer hybrid approaches can mitigate fundamental tradeoffs in sensing and processing.