Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
Conference of the European Chapter of the Association for Computational Linguistics(2024)
摘要
Topic models are a popular tool for understanding text collections, but their
evaluation has been a point of contention. Automated evaluation metrics such as
coherence are often used, however, their validity has been questioned for
neural topic models (NTMs) and can overlook a models benefits in real world
applications. To this end, we conduct the first evaluation of neural,
supervised and classical topic models in an interactive task based setting. We
combine topic models with a classifier and test their ability to help humans
conduct content analysis and document annotation. From simulated, real user and
expert pilot studies, the Contextual Neural Topic Model does the best on
cluster evaluation metrics and human evaluations; however, LDA is competitive
with two other NTMs under our simulated experiment and user study results,
contrary to what coherence scores suggest. We show that current automated
metrics do not provide a complete picture of topic modeling capabilities, but
the right choice of NTMs can be better than classical models on practical task.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要