Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States.

Determining affective states such as confusion from students' participation in online discussion forums can be useful for instructors of a large classroom. However, manual annotation of forum posts by instructors or paid crowd workers is both time-consuming and expensive. In this work, we harness affordances prevalent in social media to allow students to self-annotate their discussion posts with a set of hashtags and emojis, a process that is fast and cheap. For students, self-annotation with hashtags and emojis provides another channel for self-expression, as well as a way to signal to instructors and other students on the lookout for certain types of messages. This method also provides an easy way to acquire a labeled dataset of affective states, allowing us distinguish between more nuanced emotions such as confusion and curiosity. From a dataset of over 25,000 discussion posts from two courses containing self-annotated posts by students, we demonstrate how we can identify linguistic differences between posts expressing confusion versus curiosity, achieving 83% accuracy at distinguishing between the two affective states.