In this paper, we propose a novel information theoretic surrogate loss; normalized conditional mutual information (NCMI); as a drop in alternative to the de facto cross-entropy (CE) for training deep neural network (DNN) based classifiers. We first observe that the model's NCMI is inversely proportional to its accuracy. Building on this insight, we introduce an alternating algorithm to efficiently minimize the NCMI. Across image recognition and whole-slide imaging (WSI) subtyping benchmarks, NCMI-trained models surpass state of the art losses by substantial margins at a computational cost comparable to that of CE. Notably, on ImageNet, NCMI yields a 2.77% top-1 accuracy improvement with ResNet-50 comparing to the CE; on CAMELYON-17, replacing CE with NCMI improves the macro-F1 by 8.6% over the strongest baseline. Gains are consistent across various architectures and batch sizes, suggesting that NCMI is a practical and competitive alternative to CE.

翻译：本文提出一种新颖的信息论替代损失函数——归一化条件互信息，作为训练基于深度神经网络的分类器时，对事实上的标准交叉熵损失的直接替代方案。我们首先观察到模型的归一化条件互信息与其准确率成反比。基于这一发现，我们提出一种交替优化算法来高效最小化归一化条件互信息。在图像识别和全切片成像亚型分型基准测试中，使用归一化条件互信息训练的模型以与交叉熵相当的计算成本，在性能上显著超越现有最优损失函数。值得注意的是，在ImageNet数据集上，使用ResNet-50架构时，归一化条件互信息相比交叉熵将Top-1准确率提升了2.77%；在CAMELYON-17数据集上，用归一化条件互信息替代交叉熵，其宏平均F1分数比最强基线提升了8.6%。这些性能提升在不同网络架构和批次大小下均保持一致，表明归一化条件互信息是交叉熵的一种实用且具有竞争力的替代方案。