Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. We open-source our code and data (https://github.com/kiranpurohit/EXPLORA).

翻译：基于文本及混合来源（包括表格）回答推理型复杂问题是一项具有挑战性的任务。近期大型语言模型（LLM）的进展实现了上下文学习（ICL），使得LLM仅需少量演示样本（示例）即可掌握特定任务的能力。ICL中的一个关键挑战在于最优示例的选择，其可分为任务特定（静态）或测试样本特定（动态）两种类型。静态示例能提供更快的推理时间，并在测试样本分布上具有更强的鲁棒性。本文提出一种面向复杂推理任务的静态示例子集选择算法。我们引入EXPLORA——一种新颖的探索方法，旨在估计评分函数的参数，该函数可在不引入置信度信息的情况下评估示例子集。EXPLORA将LLM调用次数显著减少至现有最优方法的约11%，并实现了12.24%的显著性能提升。我们已开源代码与数据（https://github.com/kiranpurohit/EXPLORA）。