Toward Artificial Emotional Intelligence for Cooperative Social Human–Machine Interaction

The aptitude to identify the emotional states of others and response to exposed emotions is an important aspect of human social intelligence. Robots are expected to be prevalent in society to assist humans in various tasks. Human–robot interaction (HRI) is of critical importance in the assistive robotics sector. Smart digital assistants and assistive robots fail quite often when a request is not well defined verbally. When the assistant fails to provide services as desired, the person may exhibit an emotional response such as anger or frustration through expressions in their face and voice. It is critical that robots understand not only the language, but also human psychology. A novel affection-based perception architecture for cooperative HRIs is studied in this paper, where the agent is expected to recognize human emotional states, thus encourages a natural bonding between the human and the robotic artifact. We propose a method to close the loop using measured emotions to grade HRIs. This metric will be used as a reward mechanism to adjust the assistant’s behavior adaptively. Emotion levels from users are detected through vision and speech inputs processed by deep neural networks (NNs). Negative emotions exhibit a change in performance until the user is satisfied.