All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

翻译：所有基于文本的语言问题均可归结为生成或嵌入两类任务。当前模型仅在单一任务上表现优异。我们提出生成式表征指令微调（GRIT）方法，使大型语言模型能够通过指令区分生成式任务与嵌入任务，从而同步处理两类任务。相较于其他开源模型，我们训练的GritLM 7B在大型文本嵌入基准（MTEB）上创下新纪录，并在多种生成式任务中超越所有同等规模模型。通过进一步扩展，GritLM 8x7B在保持顶尖嵌入模型性能的同时，优于我们尝试过的所有开源生成式语言模型。值得注意的是，GRIT的训练效果与单独使用生成式或嵌入数据训练相当，因此我们可以在不损失性能的前提下统一两类任务。除其他优势外，通过GRIT实现的统一机制可将长文档的检索增强生成（RAG）速度提升60%以上，无需再分别部署检索模型与生成模型。模型、代码等资源已开源至https://github.com/ContextualAI/gritlm。