Supervised contrastive loss (SCL) is a competitive and often superior alternative to the cross-entropy loss for classification. While prior studies have demonstrated that both losses yield symmetric training representations under balanced data, this symmetry breaks under class imbalances. This paper presents an intriguing discovery: the introduction of a ReLU activation at the final layer effectively restores the symmetry in SCL-learned representations. We arrive at this finding analytically, by establishing that the global minimizers of an unconstrained features model with SCL loss and entry-wise non-negativity constraints form an orthogonal frame. Extensive experiments conducted across various datasets, architectures, and imbalance scenarios corroborate our finding. Importantly, our experiments reveal that the inclusion of the ReLU activation restores symmetry without compromising test accuracy. This constitutes the first geometry characterization of SCL under imbalances. Additionally, our analysis and experiments underscore the pivotal role of batch selection strategies in representation geometry. By proving necessary and sufficient conditions for mini-batch choices that ensure invariant symmetric representations, we introduce batch-binding as an efficient strategy that guarantees these conditions hold.

翻译：监督对比损失（SCL）是分类任务中交叉熵损失的一种具有竞争力且通常更优的替代方案。尽管先前研究表明，在平衡数据下两种损失均能产生对称的训练表示，但这种对称性在类别不平衡时会遭到破坏。本文提出一个引人注目的发现：在最终层引入ReLU激活函数可有效恢复SCL学习表示的对称性。我们通过严格分析得出该结论——证明具有SCL损失与逐元素非负约束的无约束特征模型的全局最小值构成正交框架。跨多种数据集、架构和不平衡场景的大量实验验证了该发现。重要的是，实验表明ReLU激活的引入在恢复对称性的同时不会降低测试精度。这构成了不平衡条件下SCL表示的首次几何特征刻画。此外，我们的分析与实验强调了批选择策略在表示几何中的关键作用。通过证明保证不变对称表示的mini-batch选择的充要条件，我们提出批绑定（batch-binding）作为确保这些条件成立的高效策略。