VirTeach: mmWave Radar Point Cloud Based Pose Estimation With Virtual Data as a Teacher

This paper presents VirTeach, the exploitation of using the virtual point cloud (VPC) as an assisted teacher in the learning process for human pose estimation incorporated with the millimeter wave (mmWave) radar point cloud (RPC). Due to the observations that the involvement of different body parts varies in moving ranges and directions while performing postures and mmWave signals possess inherent characteristics (i.e., specular reflection, radical sensitiveness) during their perception, the RPC data suffer from the issues of blind spots and data imbalance for the response points induced by specific joints, leading to insufficient and biased learning and thus large estimation errors for them. To address these issues, we introduce the VPC data driven by real human motions to assist the learning process, which is indispensable in explicitly imposing task-specific constraints for the distorted RPC data in a fashion of learning by teaching. Specifically, we first design a generation module to produce the desired VPC data considering both the global structure and local motions of the human skeleton, serving as the “teacher” to augment the corrupted RPC data. Secondly, we incorporate the global and local guidance from the VPC data within a coarse-to-fine pose estimation framework. The former addresses the blind spots issue by completing the RPC data to facilitate the global skeleton reconstruction, while the latter is targeted for strengthening the contribution of specific joints through constructing the local spatial-temporal neighborhood to further refine their positions. Extensive experiments are conducted to validate the effectiveness of the proposed method.