Hierarchical Attention Approach in Multimodal Emotion Recognition for Human Robot Interaction

The ability to perceive human emotions is one of the key elements that may promise a natural, genuine and more reliable human robot interaction. Though emotional perception in human robot interaction has been challenged by many difficulties, the lack of contextual understanding is can be attributed as the biggest hurdle in this regard. Most of the literature refers to the datasets developed in controlled environment to validate the performance of their systems which happens to be really good. Still those systems are very far from achieving that kind of performance in real-life scenarios. In this paper a multimodal emotion recognition strategy is presented, that uses voice features in addition to the facial expressions to determine the emotional state of the user. A hierarchical attention layer is used to for feature fusion purpose. The final system is end-to-end trainable, multimodal approach makes it more resilient to the changes in environment. The achieved 76.3% accuracy for eNterface05 video dataset, which is higher than the any single modality approach in the comparison.