Blending Task Success and User Satisfaction: Analysis of Learned Dialogue Behaviour with Multiple Rewards

Recently, principal reward components for dialogue policy reinforcement learning use task success and user satisfaction independently and neither the resulting learned behaviour has been analysed nor a suitable proper analysis method even existed. In this work, we employ both principal reward components jointly and propose a method to analyse the resulting behaviour through a structured way of probing the learned policy. We show that blending both reward components increases user satisfaction without sacrificing task success even in more hostile environments and provide insight about actions chosen by the learned policies.