Cost-Sensitive Multi-strategy Active Annotation for Text Classification

Different parts of an instance may be strong or weak indicators of the instance's label. We propose a new annotation strategy, where in addition to an instance's label, the annotator indicates parts of the instance that are rationales for its label. For two text classification tasks, we show that rationales provide a significant improvement in performance. Each instance (with or without rationales) may provide different incremental value to the learning algorithm. Annotation cost may also vary across instances and annotation strategies. We propose a cost-sensitive active learning approach for joint selection of instance and strategy that automatically determines which instances to query and whether to ask for rationales. We show that the proposed approach outperforms instance selection with a fixed strategy. While the additional cost for rationales may vary across annotators, user interface design, task, etc. We show that the proposed approach is able to select the right annotation strategy for each scenario.