D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
CoRR(2024)
摘要
While human annotations play a crucial role in language technologies,
annotator subjectivity has long been overlooked in data collection. Recent
studies that have critically examined this issue are often situated in the
Western context, and solely document differences across age, gender, or racial
groups. As a result, NLP research on subjectivity have overlooked the fact that
individuals within demographic groups may hold diverse values, which can
influence their perceptions beyond their group norms. To effectively
incorporate these considerations into NLP pipelines, we need datasets with
extensive parallel annotations from various social and cultural groups. In this
paper we introduce the dataset: a large-scale cross-cultural dataset
of parallel annotations for offensive language in over 4.5K sentences annotated
by a pool of over 4k annotators, balanced across gender and age, from across 21
countries, representing eight geo-cultural regions. The dataset contains
annotators' moral values captured along six moral foundations: care, equality,
proportionality, authority, loyalty, and purity. Our analyses reveal
substantial regional variations in annotators' perceptions that are shaped by
individual moral values, offering crucial insights for building pluralistic,
culturally sensitive NLP models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要