A Taxonomy of Methods, Tools, and Approaches for Enabling Collaborative Annotation.

Collaborative annotation reduces the time and cost of the annotation task and increases data quality. For knowledge-intensive contexts, collaborative annotation is an interesting approach to combine expert users’ knowledge with domain data. This work was motivated by a use case where expert users’ tacit knowledge associated with externalized data can produce new domain knowledge. That new knowledge can inform AI-based systems and make them more accurate for that specific domain. To identify and understand the collaborative annotation process’ dimensions, especially when the annotation is used to support AI models, we defined the following research question: “What are the methods, tools, and solutions to allow collaborative annotation?”. We did a systematic literature review, using five specific questions to help us answer the main research question. Our findings allowed us to identify different domains that adopt this approach, tools for collaborative annotation, players present in the process, and strategies for collaborative annotation. We summarize the findings in a taxonomy to represent aspects related to the characterization of annotation, the collaboration itself, and user experience strategies to support this process. We conclude the paper suggesting open opportunities for future research and implications for the design of an AI-based system to support experts in knowledge-intensive processes.