Large language models (LLMs) have the remarkable ability to solve new tasks with just a few examples, but they need access to the right tools. Retrieval Augmented Generation (RAG) addresses this problem by retrieving a list of relevant tools for a given task. However, RAG's tool retrieval step requires all the required information to be explicitly present in the query. This is a limitation, as semantic search, the widely adopted tool retrieval method, can fail when the query is incomplete or lacks context. To address this limitation, we propose Context Tuning for RAG, which employs a smart context retrieval system to fetch relevant information that improves both tool retrieval and plan generation. Our lightweight context retrieval model uses numerical, categorical, and habitual usage signals to retrieve and rank context items. Our empirical results demonstrate that context tuning significantly enhances semantic search, achieving a 3.5-fold and 1.5-fold improvement in Recall@K for context retrieval and tool retrieval tasks respectively, and resulting in an 11.6% increase in LLM-based planner accuracy. Additionally, we show that our proposed lightweight model using Reciprocal Rank Fusion (RRF) with LambdaMART outperforms GPT-4 based retrieval. Moreover, we observe context augmentation at plan generation, even after tool retrieval, reduces hallucination.

翻译：大型语言模型（LLM）具有仅凭少量示例即可解决新任务的卓越能力，但需要访问正确的工具。检索增强生成（RAG）通过为给定任务检索相关工具列表来解决此问题。然而，RAG的工具检索步骤要求查询中显式包含所有必要信息。这一局限性导致当查询不完整或缺乏上下文时，广泛采用基于语义搜索的工具检索方法可能失效。为克服此局限，我们提出面向RAG的上下文调优方法，该方法采用智能上下文检索系统获取相关信息，从而同时提升工具检索与计划生成性能。我们提出的轻量级上下文检索模型利用数值型、类别型及习惯性使用信号来检索并排序上下文项。实验结果表明，上下文调优显著增强了语义搜索：在上下文检索与工具检索任务中，Recall@K分别提升3.5倍与1.5倍，并使基于LLM的计划生成准确率提高11.6%。此外，我们证明采用LambdaMART的互惠排名融合（RRF）的轻量级模型性能优于基于GPT-4的检索。同时观察到，即使在工具检索之后，在计划生成阶段进行上下文增强仍能减少幻觉现象。