In this paper, we propose a novel information theoretic surrogate loss; normalized conditional mutual information (NCMI); as a drop in alternative to the de facto cross-entropy (CE) for training deep neural network (DNN) based classifiers. We first observe that the model's NCMI is inversely proportional to its accuracy. Building on this insight, we introduce an alternating algorithm to efficiently minimize the NCMI. Across image recognition and whole-slide imaging (WSI) subtyping benchmarks, NCMI-trained models surpass state of the art losses by substantial margins at a computational cost comparable to that of CE. Notably, on ImageNet, NCMI yields a 2.77% top-1 accuracy improvement with ResNet-50 comparing to the CE; on CAMELYON-17, replacing CE with NCMI improves the macro-F1 by 8.6% over the strongest baseline. Gains are consistent across various architectures and batch sizes, suggesting that NCMI is a practical and competitive alternative to CE.

翻译：本文提出一种新颖的信息论替代损失函数——归一化条件互信息（NCMI），作为训练基于深度神经网络（DNN）分类器时对事实标准交叉熵（CE）的直接替代方案。我们首先观察到模型的NCMI与其准确率呈反比关系。基于这一发现，我们提出一种交替优化算法以高效最小化NCMI。在图像识别和全切片成像（WSI）亚型分型基准测试中，经NCMI训练的模型以与CE相当的计算成本，显著超越了现有最优损失函数的性能。值得注意的是，在ImageNet数据集上，使用ResNet-50架构时NCMI相比CE实现了2.77%的top-1准确率提升；在CAMELYON-17数据集上，用NCMI替代CE使得宏F1分数较最强基线提升了8.6%。这些增益在不同网络架构和批次规模下均保持一致性，表明NCMI是CE的一种实用且具有竞争力的替代方案。