Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we investigate the memorization properties of SimCLR, a widely used contrastive self-supervised learning approach, and compare them to the memorization of supervised learning and random labels training. We find that both training objects and augmentations may have different complexity in the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar to random labels training in terms of the distribution of training objects complexity.

翻译：对深层神经网络(DNN)的记忆研究有助于了解DNN学习的模式和方式,并激励对DNN培训方法的改进。在这项工作中,我们调查了SimCLR的记忆特性,这是一个广泛使用的对比式自我监督学习方法,并把它们与监督学习和随机标签培训的记忆化进行比较。我们发现,在SimCLR如何了解这些模式和方式的意义上,培训对象和扩增可能具有不同的复杂性。此外,我们表明,SimCLR在培训对象复杂性的分布方面类似于随机标签培训。