Contrastive learning is a self-supervised representation learning framework, where two positive views generated through data augmentation are made similar by an attraction force in a data representation space, while a repulsive force makes them far from negative examples. Non-contrastive learning, represented by BYOL and SimSiam, further gets rid of negative examples and improves computational efficiency. While learned representations may collapse into a single point due to the lack of the repulsive force at first sight, Tian et al. (2021) revealed through the learning dynamics analysis that the representations can avoid collapse if data augmentation is sufficiently stronger than regularization. However, their analysis does not take into account commonly-used feature normalization, a normalizer before measuring the similarity of representations, and hence excessively strong regularization may collapse the dynamics, which is an unnatural behavior under the presence of feature normalization. Therefore, we extend the previous theory based on the L2 loss by considering the cosine loss, which involves feature normalization. We show that the cosine loss induces sixth-order dynamics (while the L2 loss induces a third-order one), in which a stable equilibrium dynamically emerges even if there are only collapsed solutions with given initial parameters. Thus, we offer a new understanding that feature normalization plays an important role in robustly preventing the dynamics collapse.

翻译：对比学习是一种自监督表示学习框架，其中通过数据增强生成的两个正视图在数据表示空间中通过吸引力变得相似，而排斥力则使其远离负样本。以BYOL和SimSiam为代表的非对比学习进一步摒弃了负样本，并提高了计算效率。虽然最初由于缺乏排斥力，学习到的表示可能坍缩为单一节点，但Tian等人（2021）通过动力学分析揭示，若数据增强强度显著大于正则化强度，则表示可以避免崩溃。然而，他们的分析未考虑广泛使用的特征归一化（即在测量表示相似性之前的归一化操作），因此过强的正则化可能导致动力学崩溃——这在使用特征归一化的情况下是一种非自然行为。为此，我们基于余弦损失（涉及特征归一化）扩展了先前基于L2损失的理论。研究表明，余弦损失会引发六阶动力学（而L2损失引发三阶动力学），其中即使初始参数仅存在崩溃解，也会动态涌现出稳定平衡态。因此，我们提出新见解：特征归一化在稳健防止动力学崩溃中起关键作用。