Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization.

翻译：针对特定任务优化检索增强生成（RAG）配置是一项复杂且资源密集的挑战。受此挑战驱动，RAG超参数优化（HPO）框架近期相继涌现，但其有效性尚未经过严格基准测试。为填补这一空白，我们开展了一项综合研究，在涵盖多领域的五个数据集（包括新构建的真实世界产品文档数据集）上对五种HPO算法进行了评估。本研究探索了迄今为止最大的RAG HPO搜索空间（包含全网格搜索评估），并采用三种评估指标作为优化目标。结果分析表明：RAG HPO可通过贪心法或随机搜索高效实现，且能显著提升所有数据集的RAG性能；对于贪心HPO方法，我们证明优先优化模型选择的策略优于优化过程中遵循RAG流水线顺序的常规做法。