The annotation of domain experts is important for some medical applications where the objective ground truth is ambiguous to define, e.g., the rehabilitation for some chronic diseases, and the prescreening of some musculoskeletal abnormalities without further medical examinations. However, improper uses of the annotations may hinder developing reliable models. On one hand, forcing the use of a single ground truth generated from multiple annotations is less informative for the modeling. On the other hand, feeding the model with all the annotations without proper regularization is noisy given existing disagreements. For such issues, we propose a novel Learning to Agreement (Learn2Agree) framework to tackle the challenge of learning from multiple annotators without objective ground truth. The framework has two streams, with one stream fitting with the multiple annotators and the other stream learning agreement information between annotators. In particular, the agreement learning stream produces regularization information to the classifier stream, tuning its decision to be better in line with the agreement between annotators. The proposed method can be easily added to existing backbones, with experiments on two medical datasets showed better agreement levels with annotators.

翻译：对于某些客观真值难以定义的医学应用（例如慢性疾病康复评估、无需进一步医学检查的肌肉骨骼异常预筛查），领域专家的标注至关重要。然而，不当使用标注可能阻碍可靠模型的开发。一方面，强制使用由多个标注生成的单一真值会削弱建模的信息量；另一方面，在缺乏适当正则化的情况下将所有标注输入模型，会因现有分歧而产生噪声。针对上述问题，我们提出了一种新型的"学习达成共识"（Learn2Agree）框架，旨在解决无客观真值时从多标注者学习数据的挑战。该框架包含两条分支：一条分支拟合多个标注者的意见，另一条分支学习标注者之间的共识信息。具体而言，共识学习分支为分类器分支提供正则化信号，引导其决策更好地契合标注者间的共识。所提方法可便捷地集成至现有骨干网络，在两类医学数据集上的实验表明，该方法能实现与标注者更高水平的共识匹配。