Chatbots as Advisers: the Effects of Response Variability and Reply Suggestion Buttons

As chatbots gain popularity across a variety of applications, from investment to health, they employ an increasing number of features that can influence the perception of the system. Since chatbots often provide advice or guidance, we ask: do these aspects affect the user's decision to follow their advice? We focus on two chatbot features that can influence user perception: 1) response variability in answers and delays and 2) reply suggestion buttons. We report on a between-subject study where participants made investment decisions on a simulated social trading platform by interacting with a chatbot providing advice. Performance-based study incentives made the consequences of following the advice tangible to participants. We measured how often and to what extent participants followed the chatbot's advice compared to an alternative source of information. Results indicate that both response variability and reply suggestion buttons significantly increased the inclination to follow the advice of the chatbot.