Contrastive learning has emerged as a cornerstone in recent achievements of unsupervised representation learning. Its primary paradigm involves an instance discrimination task with a mutual information loss. The loss is known as InfoNCE and it has yielded vital insights into contrastive learning through the lens of mutual information analysis. However, the estimation of mutual information can prove challenging, creating a gap between the elegance of its mathematical foundation and the complexity of its estimation. As a result, drawing rigorous insights or conclusions from mutual information analysis becomes intricate. In this study, we introduce three novel methods and a few related theorems, aimed at enhancing the rigor of mutual information analysis. Despite their simplicity, these methods can carry substantial utility. Leveraging these approaches, we reassess three instances of contrastive learning analysis, illustrating their capacity to facilitate deeper comprehension or to rectify pre-existing misconceptions. Specifically, we investigate small batch size, mutual information as a measure, and the InfoMin principle.

翻译：对比学习已成为无监督表示学习近期成果的基石。其核心范式涉及一项基于互信息损失的实例判别任务，该损失函数被称为 InfoNCE，通过互信息分析的角度为对比学习提供了关键洞见。然而，互信息的估计可能具有挑战性，这造成了其数学基础的优雅性与估计复杂性之间的鸿沟。因此，从互信息分析中得出严谨的洞见或结论变得错综复杂。在本研究中，我们引入了三种新方法及若干相关定理，旨在提升互信息分析的严格性。这些方法尽管简单，但能承载显著实用性。借助这些方法，我们重新审视了三个对比学习分析实例，展示了它们促进更深入理解或纠正既有误解的能力。具体而言，我们探讨了小批量尺寸、互信息作为度量标准以及 InfoMin 原则这三个方面。