Data annotated by humans is a source of knowledge by describing the peculiarities of the problem and therefore fueling the decision process of the trained model. Unfortunately, the annotation process for subjective natural language processing (NLP) problems like offensiveness or emotion detection is often very expensive and time-consuming. One of the inevitable risks is to spend some of the funds and annotator effort on annotations that do not provide any additional knowledge about the specific task. To minimize these costs, we propose a new model-based approach that allows the selection of tasks annotated individually for each text in a multi-task scenario. The experiments carried out on three datasets, dozens of NLP tasks, and thousands of annotations show that our method allows up to 40% reduction in the number of annotations with negligible loss of knowledge. The results also emphasize the need to collect a diverse amount of data required to efficiently train a model, depending on the subjectivity of the annotation task. We also focused on measuring the relation between subjective tasks by evaluating the model in single-task and multi-task scenarios. Moreover, for some datasets, training only on the labels predicted by our model improved the efficiency of task selection as a self-supervised learning regularization technique.

翻译：人工标注的数据通过描述问题的特殊性，为训练模型的决策过程提供知识来源。然而，对于冒犯性检测或情感识别等主观自然语言处理（NLP）问题，标注过程往往成本高昂且耗时巨大。其中不可避免的风险之一，是将部分资金和标注者的精力浪费在无法为特定任务提供额外知识的标注上。为最小化这些成本，我们提出一种新的基于模型的方法，该方法允许在多任务场景中为每段文本独立选择需标注的任务。在三个数据集、数十个NLP任务及数千条标注上进行的实验表明，我们的方法能在知识损失可忽略不计的前提下，将标注数量减少高达40%。结果同时强调，根据标注任务的主观性程度，需要收集多样化的数据来高效训练模型。我们还通过评估单任务与多任务场景下的模型表现，重点衡量了主观任务间的关联性。此外，针对某些数据集，仅利用我们模型预测的标签进行训练，可作为自监督学习正则化技术提升任务选择效率。