Traditional sentence embedding methods employ token-level contrastive learning on non-generative pre-trained models. Recently, there have emerged embedding methods based on generative large language models (LLMs). These methods either rely on fixed prompt templates or involve modifications to the model architecture. The former lacks further optimization of the model and results in limited performance, while the latter alters the internal computational mechanisms of the model, thereby compromising its generative capabilities. We propose SemPA, a novel approach that boosts the sentence representations while preserving the generative ability of LLMs via semantic preference alignment. We leverage sentence-level Direct Preference Optimization (DPO) to efficiently optimize LLMs on a paraphrase generation task, where the model learns to discriminate semantically equivalent sentences while preserving inherent generative capacity. Theoretically, we establish a formal connection between DPO and contrastive learning under the Plackett-Luce model framework. Empirically, experimental results on both semantic textual similarity tasks and various benchmarks for LLMs show that SemPA achieves better semantic representations without sacrificing the inherent generation capability of LLMs.

翻译：传统句子嵌入方法在非生成式预训练模型上采用词元级对比学习。近年来，出现了基于生成式大语言模型（LLMs）的嵌入方法。这些方法要么依赖固定提示模板，要么涉及对模型架构的修改。前者缺乏对模型的进一步优化，导致性能受限；而后者改变了模型的内部计算机制，从而损害其生成能力。我们提出SemPA，一种通过语义偏好对齐来增强句子表示同时保持LLMs生成能力的新方法。我们利用句子级直接偏好优化（DPO）在释义生成任务上高效优化LLMs，使模型学会区分语义等价的句子，同时保留固有的生成能力。理论上，我们在Plackett-Luce模型框架下建立了DPO与对比学习的形式化关联。实证结果表明，在语义文本相似性任务和多项LLMs基准测试中，SemPA在不牺牲LLMs固有生成能力的前提下获得了更优的语义表示。