Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.

翻译：白化损失为基于联合嵌入架构的自监督学习提供了防止特征坍塌的理论保证。通常，该方法采用硬白化策略，对嵌入向量进行变换并对白化输出施加损失函数。本研究提出谱变换框架，通过调整嵌入向量的谱分布，探索除白化之外能够避免维度坍塌的函数形式。我们证明白化是ST的一种特例，实验发现表明其他ST实例同样具备防止坍塌的能力。此外，我们提出一种新的ST实例——基于迹损失的迭代归一化方法。理论分析证实INTL能有效防止坍塌，并在优化过程中将嵌入谱调节为等特征值分布。在ImageNet分类和COCO目标检测上的实验表明，INTL在学习更优表征方面具有潜力。代码开源地址：https://github.com/winci-ai/INTL。