In this work, we aim to connect two research areas: instruction models and retrieval-based models. While instruction-tuned Large Language Models (LLMs) excel at extracting information from text, they are not suitable for semantic retrieval. Similarity search over embedding vectors allows to index and query vectors, but the similarity reflected in the embedding is sub-optimal for many use cases. We identify the task of retrieving sentences based on abstract descriptions of their content. We demonstrate the inadequacy of current text embeddings and propose an alternative model that significantly improves when used in standard nearest neighbor search. The model is trained using positive and negative pairs sourced through prompting an a large language model (LLM). While it is easy to source the training material from an LLM, the retrieval task cannot be performed by the LLM directly. This demonstrates that data from LLMs can be used not only for distilling more efficient specialized models than the original LLM, but also for creating new capabilities not immediately possible using the original model.

翻译：在本工作中，我们旨在连接两个研究领域：指令模型与基于检索的模型。尽管经过指令微调的大型语言模型（LLMs）擅长从文本中提取信息，但不宜用于语义检索。基于嵌入向量的相似性搜索虽能实现向量的索引与查询，但嵌入所反映的相似性在许多应用场景中并非最优。我们明确了基于内容抽象描述检索句子的任务，论证了当前文本嵌入的不足，并提出一种替代模型，该模型在标准最近邻搜索中性能显著提升。该模型通过利用大型语言模型（LLM）提示生成的正面与负面配对进行训练。虽然从LLM获取训练材料较为容易，但检索任务无法直接由LLM完成。这表明，LLM生成的数据不仅可用于蒸馏出比原始LLM更高效的专用模型，还可用于创造原始模型无法直接实现的新能力。