Recent studies have proposed using large language models (LLMs) for sentence embeddings. However, most existing LLMs are built with an autoregressive architecture that primarily captures forward dependencies while neglecting backward dependencies. Previous work has highlighted the importance of backward dependencies in improving sentence embeddings. To address this issue, in this paper, we first present quantitative evidence demonstrating the limited learning of backward dependencies in LLMs. Then, we propose a novel approach called Dependency-Enhanced Large Language Model (DeeLM) to improve sentence embeddings. Specifically, we found a turning point in LLMs, where surpassing specific LLM layers leads to a significant performance drop in the semantic textual similarity (STS) task. STS is a crucial task for evaluating sentence embeddings. We then extract the layers after the turning point to make them bidirectional, allowing for the learning of backward dependencies. Extensive experiments demonstrate that DeeLM outperforms baselines and achieves state-of-the-art performance across various STS tasks.

翻译：近期研究提出使用大语言模型（LLMs）生成句子嵌入。然而，现有LLMs大多采用自回归架构，主要捕获前向依赖关系而忽略后向依赖关系。先前工作已强调后向依赖关系对改进句子嵌入的重要性。为解决该问题，本文首先通过量化证据展示LLMs对后向依赖关系的学习局限性。随后，我们提出一种名为依赖增强型大语言模型（DeeLM）的新方法用于改进句子嵌入。具体而言，我们发现LLM中存在一个转折点——当超过特定LLM层后，语义文本相似度（STS）任务的性能会显著下降。STS是评估句子嵌入的关键任务。我们提取转折点后的层进行双向化改造，使其能够学习后向依赖关系。大量实验表明，DeeLM在多项STS任务中均优于基线方法，并达到当前最优性能。