Predicting Sentiment Toward Transportation In Social Media Using Visual And Textual Features

Social media platforms can be used by transportation agencies to receive feedback from their customers, thus creating two-way communication between the service provider and its consumers. Sentiment analysis is one method of aggregating overall polarity (positive or negative) towards a topic. However, most sentiment analysis methods rely on text processing, thus ignoring the large amount of image data present in popular social networks. The primary aim of this study is to exploit image data in conjunction with text and to evaluate this integrated approach for sentiment analysis for transportation. This study used image, captions, and comments data from the Instagram social network that were marked as being relevant to California Department of Transportation (Caltrans) and attempted to predict the expressed sentiment towards this agency. A set of high-level features were extracted from images using the web-based Microsoft Cognitive Services APIs. These features included the detection of faces and 86 categories which describe the images. Text features included the set of individual words and structural features. The experiment results of different machine learning techniques show a gain in precision when images and texts are combined compared to text-only approaches, thus confirming the relevance of visual content usage. The precision reaches a performance close to human classification agreement (typically approximately 80%). However, the results do not indicate that visual features are more informative than text features.