Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 $\approx 0.8$ for scenarios with prominent gradients in search space, using only $\sim20\%$ of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.

翻译：大型语言模型的最新进展正在改变机器学习/人工智能的开发范式，这促使我们需要重新审视面向检索增强生成系统的自动化机器学习原则。为应对RAG系统中超参数优化与在线适应的挑战，本文提出AutoRAG-HP框架，将超参数调优问题建模为在线多臂老虎机问题，并引入一种新颖的双层层次化多臂老虎机方法，以实现对庞大搜索空间的高效探索。我们基于ALCE-ASQA和Natural Questions数据集，对top-k检索文档数、提示压缩比和嵌入方法等超参数进行了大规模调优实验。通过对三个超参数进行联合优化的评估表明：在搜索空间梯度显著的应用场景中，基于MAB的在线学习方法仅需网格搜索法约20%的大语言模型API调用量，即可实现Recall@5 $\approx 0.8$的性能指标。此外，在更具挑战性的优化场景中，所提出的Hier-MAB方法均优于其他基线模型。相关代码将在https://aka.ms/autorag 公开。