Self-training is a well-known approach for semi-supervised learning. It consists of iteratively assigning pseudo-labels to unlabeled data for which the model is confident and treating them as labeled examples. For neural networks, softmax prediction probabilities are often used as a confidence measure, despite the fact that they are known to be overconfident, even for wrong predictions. This phenomenon is particularly intensified in the presence of sample selection bias, i.e., when data labeling is subject to some constraint. To address this issue, we propose a novel confidence measure, called $\mathcal{T}$-similarity, built upon the prediction diversity of an ensemble of linear classifiers. We provide the theoretical analysis of our approach by studying stationary points and describing the relationship between the diversity of the individual members and their performance. We empirically demonstrate the benefit of our confidence measure for three different pseudo-labeling policies on classification datasets of various data modalities.

翻译：自训练是半监督学习中一种广为人知的方法。它通过迭代方式为模型有信心的未标注数据分配伪标签，并将其视为标注样本。对于神经网络而言，尽管已知softmax预测概率即使对错误预测也会过度自信，但其仍常被用作置信度度量。这种现象在样本选择偏差存在时尤为加剧，即数据标注受到某些约束限制。为解决这一问题，我们提出了一种基于线性分类器集成预测多样性的新置信度度量，称为$\mathcal{T}$-相似性。我们通过研究平稳点并描述各成员多样性与其性能之间的关系，对方法进行了理论分析。我们通过在多种数据模态的分类数据集上采用三种不同伪标签策略，实验证明了该置信度度量的有效性。