Multi-path Convolutional Neural Network based on Rectangular Kernel with Path Signature Features for Gesture Recognition

Skeleton based gesture recognition has gained more attention due to its wide application and large-scale databases availability. Recent methods designed for skeleton sequence data mainly pay attention to network architecture but ignore an essential characteristic of skeleton sequences that the temporal dimensionality of skeleton sequences is usually higher than its spatial dimensionality. Directly applying CNNs designed for image classification to skeleton-based data can not capture this unique property. Considering this fact, we propose the rectangular convolution and pooling to skeleton sequence data. Temporal features are crucial for gesture action recognition. Further, we introduce path signature features (PSF) to represent temporal variation characteristics of each joint. Moreover, there only exist a few minor distinctions between some gestures. To classify them more accurately, we add two sub-networks to extract discriminative features from two hands respectively. We evaluate our method on three major benchmark gesture datasets, i.e., ChaLearn 2013, ChaLearn 2016 and MSRC-12, and reach the state-of-the-art performance.