The performance of sentence encoders can be significantly improved through the simple practice of fine-tuning using contrastive loss. A natural question arises: what characteristics do models acquire during contrastive learning? This paper theoretically and experimentally shows that contrastive-based sentence encoders implicitly weight words based on information-theoretic quantities; that is, more informative words receive greater weight, while others receive less. The theory states that, in the lower bound of the optimal value of the contrastive learning objective, the norm of word embedding reflects the information gain associated with the distribution of surrounding words. We also conduct comprehensive experiments using various models, multiple datasets, two methods to measure the implicit weighting of models (Integrated Gradients and SHAP), and two information-theoretic quantities (information gain and self-information). The results provide empirical evidence that contrastive fine-tuning emphasizes informative words.

翻译：句子编码器的性能可以通过简单使用对比损失进行微调而显著提升。一个自然的问题随之产生：在对比学习过程中，模型获得了哪些特征？本文从理论和实验两方面表明，基于对比学习的句子编码器会根据信息论量对词汇进行隐式加权，即信息量更大的词汇获得更大权重，而其他词汇权重较小。理论指出，在对比学习目标函数最优值的下界中，词嵌入的范数反映了与周围词汇分布相关的信息增益。我们还使用多种模型、多个数据集、两种衡量模型隐式加权的方法（集成梯度和SHAP）以及两种信息论量（信息增益和自信息）进行了全面实验。结果提供了实证证据，证明对比微调会强调信息性词汇。