In this paper, we propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. SSL approaches typically work through some invariance term to ensure consistency between similar samples and a regularization term to prevent global dimensional collapse. Dimensional collapse refers to data representations spanning a lower-dimensional subspace. Recent work has demonstrated that the representation space of these algorithms gradually reflects a semantic hierarchical structure as training progresses. Data samples of the same hierarchical grouping tend to exhibit greater dimensional collapse locally compared to the dataset as a whole due to sharing features in common with each other. Ideally, SSL algorithms would take advantage of this hierarchical emergence to have an additional regularization term to account for this local dimensional collapse effect. However, the construction of existing SSL algorithms does not account for this property. To address this, we propose an adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively. This decomposition is based on an adaptive threshold that gradually lowers to reflect the emerging hierarchical structure of the representation space throughout training. It is based on an analysis of the cosine similarity distribution of samples in a batch. We demonstrate that this hierarchical emergence exploitation (HEX) approach can be integrated across a wide variety of SSL algorithms. Empirically, we show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.

翻译：本文提出一种算法，可应用于多种自监督学习方法之上，以利用训练过程中涌现的层次结构。自监督学习方法通常通过不变性项确保相似样本间的一致性，并通过正则化项防止全局维度坍缩。维度坍缩指数据表征仅分布在低维子空间中的现象。近期研究表明，随着训练进行，这些算法的表征空间会逐渐反映语义层次结构。由于共享共同特征，同层次分组的数据样本相较于整体数据集往往表现出更显著的局部维度坍缩。理想情况下，自监督算法应利用这种层次涌现特性，通过附加正则化项处理局部维度坍缩效应。然而现有自监督算法的构建并未考虑此特性。为此，我们提出一种自适应算法，将InfoNCE损失函数的分母加权分解为两个项：局部层次正则化项与全局坍缩正则化项。该分解基于自适应阈值实现，该阈值随训练进程逐步降低以反映表征空间涌现的层次结构，其确定依据为批次样本余弦相似度分布的分析结果。我们证明这种层次涌现利用方法可与多种自监督算法集成。实验表明，在ImageNet数据集上经过100轮训练后，该方法相较于基线自监督方法在分类准确率上最高可获得5.6%的相对提升。