Self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. One straightforward approach is to finetune a pretrained language model (PLM) with a representation learning method such as contrastive learning. While this approach achieves impressive performance on larger PLMs, the performance rapidly degrades as the number of parameters decreases. In this paper, we propose a framework called Self-supervised Cross-View Training (SCT) to narrow the performance gap between large and small PLMs. To evaluate the effectiveness of SCT, we compare it to 5 baseline and state-of-the-art competitors on seven Semantic Textual Similarity (STS) benchmarks using 5 PLMs with the number of parameters ranging from 4M to 340M. The experimental results show that STC outperforms the competitors for PLMs with less than 100M parameters in 18 of 21 cases.

翻译：自监督句子表征学习旨在不依赖人工标注的情况下构建句子的嵌入空间。一种直接的方法是使用对比学习等表征学习方法微调预训练语言模型（PLM）。尽管这种方法在较大规模的PLM上取得了显著性能，但随着参数量的减少，性能会迅速下降。本文提出了一种名为自监督跨视角训练（SCT）的框架，旨在缩小大规模与小规模PLM之间的性能差距。为评估SCT的有效性，我们在七个语义文本相似度（STS）基准测试中，将SCT与5个基线及最先进的竞争方法进行了比较，使用了参数规模从4M到340M的5种PLM。实验结果表明，在参数量小于100M的PLM中，SCT在21个案例中的18个上优于现有竞争方法。