Large language models (LLMs) face significant challenges stemming from their inherent limitations in knowledge, memory, alignment, and action. These challenges cannot be addressed by LLMs alone, but should rely on assistance from the external world, such as knowledge base, memory store, demonstration examples, and tools. Retrieval augmentation stands as a vital mechanism for bridging the gap between LLMs and the external assistance. However, conventional methods encounter two pressing issues. On the one hand, the general-purpose retrievers are not properly optimized for the retrieval augmentation of LLMs. On the other hand, the task-specific retrievers lack the required versatility, hindering their performance across the diverse retrieval augmentation scenarios. In this work, we present a novel approach, the LLM-Embedder, which comprehensively supports the diverse retrieval augmentation needs of LLMs with one unified embedding model. Training such a unified model is non-trivial, as various retrieval tasks aim to capture distinct semantic relationships, often subject to mutual interference. To address this challenge, we systematically optimize our training methodology. This includes reward formulation based on LLMs' feedback, the stabilization of knowledge distillation, multi-task fine-tuning with explicit instructions, and homogeneous in-batch negative sampling. These optimization strategies contribute to the outstanding empirical performance of the LLM-Embedder. Notably, it yields remarkable enhancements in retrieval augmentation for LLMs, surpassing both general-purpose and task-specific retrievers in various evaluation scenarios. Our checkpoint and source code are publicly available at https://github.com/FlagOpen/FlagEmbedding.

翻译：大语言模型在知识、记忆、对齐和行动方面存在固有局限，面临重大挑战。这些挑战无法仅靠大语言模型解决，而应依赖外部世界的辅助，例如知识库、记忆存储、示范示例和工具。检索增强作为弥合大语言模型与外部辅助之间差距的关键机制，然而传统方法面临两大紧迫问题：一方面，通用检索器未针对大语言模型的检索增强进行适当优化；另一方面，特定任务检索器缺乏所需的通用性，阻碍了其在多样化检索增强场景中的性能。本文提出一种新方法——LLM-Embedder，通过单一统一嵌入模型全面支持大语言模型多样化的检索增强需求。训练此类统一模型并非易事，因为不同检索任务旨在捕获各异的语义关系，且常相互干扰。为应对此挑战，我们系统性地优化了训练方法，包括基于大语言模型反馈的奖励设计、知识蒸馏的稳定性优化、带显式指令的多任务微调以及同批次内齐性负采样。这些优化策略使LLM-Embedder取得了卓越的实证性能。值得注意的是，它在多种评估场景中显著提升了针对大语言模型的检索增强效果，性能超越通用检索器和特定任务检索器。我们的检查点与源代码已公开于https://github.com/FlagOpen/FlagEmbedding。