LLM-based text embedders typically encode the semantic content of their input. However, embedding tasks require mapping diverse inputs to similar outputs. Typically, this input-output is addressed by training embedding models with paired data using contrastive learning. In this work, we propose a novel self-supervised approach, LLM2Vec-Gen, which adopts a different paradigm: rather than encoding the input, we learn to represent the model's potential response. Specifically, we add trainable special tokens to the LLM's vocabulary, append them to input, and optimize them to represent the LLM's response in a fixed-length sequence. Training is guided by the LLM's own completion for the query, along with an unsupervised embedding teacher that provides distillation targets. This formulation helps to bridge the input-output gap and transfers LLM capabilities such as safety alignment and reasoning to embedding tasks. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 9.3% over the best unsupervised embedding teacher. We also observe up to 43.2% reduction in harmful content retrieval and 29.3% improvement in reasoning capabilities for embedding tasks. Finally, the learned embeddings are interpretable and can be decoded into text to reveal their semantic content.

翻译：基于大语言模型（LLM）的文本嵌入器通常对其输入的语义内容进行编码。然而，嵌入任务需要将多样化的输入映射到相似的输出。传统方法通常通过使用对比学习在配对数据上训练嵌入模型来解决这一输入-输出映射问题。本文提出一种新颖的自监督方法LLM2Vec-Gen，它采用了一种不同的范式：我们并非对输入进行编码，而是学习表示模型潜在的响应。具体而言，我们在LLM的词表中添加可训练的特殊标记，将其附加到输入后，并通过优化使这些标记以固定长度序列表示LLM的响应。训练过程由LLM自身对查询的补全结果指导，并结合一个提供蒸馏目标的非监督嵌入教师模型。这种设计有助于弥合输入-输出间的差距，并将LLM的安全对齐、推理等能力迁移至嵌入任务。关键的是，LLM主干网络始终保持冻结状态，且训练仅需未标注的查询数据。LLM2Vec-Gen在Massive Text Embedding Benchmark（MTEB）上实现了最先进的非监督性能，较最佳非监督嵌入教师模型提升9.3%。我们还观察到嵌入任务在有害内容检索方面最高减少43.2%，推理能力提升29.3%。最后，学习得到的嵌入具有可解释性，可通过解码为文本来揭示其语义内容。