My research explores two questions: What are the underlying principles of humans understanding conversation context as well as making proper responses, and how we can implement them on machine learning models? Research on this topic has to necessarily be at the intersection of Machine Learning, Natural Language Processing and Multimedia. In my lab, we are specifically interested in task-oriented dialogues, proactive conversational agents, and multimodal conversational search and recomendation as the application target.