Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content.

翻译：通过对比学习微调基于LLM的文本嵌入器，会将输入和输出映射到新的表示空间，从而丢弃LLM的输出语义。我们提出LLM2Vec-Gen，一种自监督的替代方法，它通过学习表示模型的潜在响应，直接在LLM的输出空间生成嵌入。具体而言，可训练的特殊标记被附加到输入中，并通过无监督嵌入教师和重构目标的引导进行优化，以将LLM自身的响应压缩为固定长度的嵌入。关键的是，LLM主干保持冻结，训练仅需无标签查询。LLM2Vec-Gen在大规模文本嵌入基准测试（MTEB）上实现了最先进的自监督性能，比无监督嵌入教师提升了8.8%。由于嵌入保留了LLM的响应空间语义，它们继承了安全对齐（有害内容检索减少高达22.6%）和推理能力（推理密集型检索提升高达35.6%）。最后，学习到的嵌入也是可解释的：它们可以被解码回文本以揭示其语义内容。