All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

翻译：所有基于文本的语言问题均可归结为生成或嵌入两类任务。当前模型仅能在一类任务上表现优异。我们提出生成式表征指令微调（Generative Representational Instruction Tuning, GRIT），该方法通过指令区分生成任务与嵌入任务，使大型语言模型能够同时处理这两类任务。与其他开源模型相比，我们训练的GritLM 7B在Massive Text Embedding Benchmark（MTEB）上达到新最优水平，并在多项生成任务中超越同规模所有模型。通过进一步扩展规模，GritLM 8x7B在保持顶尖嵌入模型性能的同时，超越了实验中所有开源生成式语言模型。关键发现是，GRIT在仅使用生成数据或嵌入数据训练时表现相当，因此我们能够在不损失性能的前提下统一两类任务。此外，GRIT的统一方案使长文档的检索增强生成（RAG）速度提升超过60%，无需再使用独立的检索模型与生成模型。模型、代码等资源已开源发布于 https://github.com/ContextualAI/gritlm。