Data-Driven Movement Subunit Extraction from Skeleton Information for Modeling Signs and Gestures

Sequence modeling for signs and gestures is an open research problem. In that direction, there is a sustained effort towards modeling signs and gestures as a sequence of subunits. In this paper, we develop a novel approach to infer movement subunits in a data-driven manner to model signs and gestures in the framework of hidden Markov models (HMM) given the skeleton information. This approach involves: (a) representation of position and movement information with measurement of hand positions relative to body parts (head, shoulders, hips); (b) modeling these features to infer a sign-specific left-to-right HMM; and (c) clustering the HMM states to infer states or subunits that are shared across signs and updating the HMM topology of signs. We investigate the application of the proposed approach on sign and gesture recognition tasks, specifically on Turkish signs HospiSign database and Italian gestures Chalearn 2014 task. On both databases, our studies show that, while yielding competitive systems, the proposed approach leads to a shared movement subunit representation that maintains discrimination across signs and gestures.