We propose "collision cross-entropy" as a robust alternative to the Shannon's cross-entropy in the context of self-labeled classification with posterior models. Assuming unlabeled data, self-labeling works by estimating latent pseudo-labels, categorical distributions y, that optimize some discriminative clustering criteria, e.g. "decisiveness" and "fairness". All existing self-labeled losses incorporate Shannon's cross-entropy term targeting the model prediction, softmax, at the estimated distribution y. In fact, softmax is trained to mimic the uncertainty in y exactly. Instead, we propose the negative log-likelihood of "collision" to maximize the probability of equality between two random variables represented by distributions softmax and y. We show that our loss satisfies some properties of a generalized cross-entropy. Interestingly, it agrees with the Shannon's cross-entropy for one-hot pseudo-labels y, but the training from softer labels weakens. For example, if y is a uniform distribution at some data point, it has zero contribution to the training. Our self-labeling loss combining collision cross entropy with basic clustering criteria is convex w.r.t. pseudo-labels, but non-trivial to optimize over the probability simplex. We derive a practical EM algorithm optimizing pseudo-labels y significantly faster than generic methods, e.g. the projectile gradient descent. The collision cross-entropy consistently improves the results on multiple self-labeled clustering examples using different DNNs.

翻译：本文提出“碰撞交叉熵”作为香农交叉熵在基于后验模型的自标注分类中的鲁棒替代方案。在无标注数据假设下，自标注通过估计潜在伪标签（分类分布y）来优化某些判别聚类准则（例如“决定性”和“公平性”）。现有所有自标注损失均包含以估计分布y为目标的香农交叉熵项，作用于模型预测的softmax输出。实际上，softmax被训练为精确模仿y中的不确定性。相反，我们提出“碰撞”的负对数似然，以最大化由softmax和y表示的两个随机变量相等的概率。我们证明该损失满足广义交叉熵的某些性质。有趣的是，对于独热形式的伪标签y，该损失与香农交叉熵一致，但对更软标签的训练效果会减弱。例如，若y在某个数据点处为均匀分布，其对训练贡献为零。我们的自标注损失结合碰撞交叉熵与基本聚类准则，关于伪标签是凸的，但如何在概率单纯形上优化则具有非平凡性。我们推导出一种实用的EM算法，其优化伪标签y的速度显著快于通用方法（如投影梯度下降）。采用不同深度神经网络的多组自标注聚类实验表明，碰撞交叉熵持续提升了结果。