On-Device Augmented Reality with Mobile GPUs

Reliable computer vision is a prerequisite for augmented reality (AR) applications; in particular when applied to mobile devices. Many state-of-the-art vision techniques employ deep neural networks. However, inference is a compute-intensive task and solely using mobile CPU can be difficult due to limited computing power, thermal constraints, and energy consumption. App developers and researchers have begun exploiting hardware accelerators to overcome these challenges. Recently, device manufacturers are adding neural processing units into high-end phones for on-device inference, but these account for only a small fraction of hand-held devices. In this paper, we present how we leverage the mobile GPU, a ubiquitous hardware accelerator on virtually every phone, to achieve real-time AR effects for both Android and iOS devices.