AAM based face tracking with temporal matching and face segmentation

Active Appearance Model (AAM) based face tracking has advantages of accurate alignment, high efficiency, and effectiveness for handling face deformation. However, AAM suffers from the generalization problem and has difficulties in images with cluttered backgrounds. In this paper, we introduce two novel constraints into AAM fitting to address the above problems. We first introduce a temporal matching constraint in AAM fitting. In the proposed fitting scheme, the temporal matching enforces an inter-frame local appearance constraint between frames. The resulting model takes advantage of temporal matching's good generalizability, but does not suffer from the mismatched points. To make AAM more stable for cluttered backgrounds, we introduce a color-based face segmentation as a soft constraint. Both constraints effectively improve the AAM tracker's performance, as demonstrated with experiments on various challenging real-world videos.