Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch prompting, a simple yet effective prompting approach that enables the LLM to run inference in batches, instead of one sample at a time. Our method reduces both token and time costs while retaining downstream performance. We theoretically demonstrate that under a few-shot in-context learning setting, the inference costs decrease almost inverse linearly with the number of samples in each batch. We extensively validate the effectiveness of batch prompting on ten datasets across commonsense QA, arithmetic reasoning, and NLI/NLU: batch prompting significantly~(up to 5x with six samples in batch) reduces the LLM (Codex) inference token and time costs while achieving better or comparable performance. For state-of-the-art Chat-based LLMs, e.g., GPT-3.5 and GPT-4, we show the benefits of batch prompting also hold. Further analysis shows that the number of samples in each batch and the complexity of tasks affect its performance. Moreover, batch prompting can be applied across different reasoning methods using LLMs. Our code can be found at the site https://github.com/xlang-ai/batch-prompting.

翻译：在工业界和实际应用中，使用大语言模型（LLMs）对大量样本进行推理会带来较高的计算成本和经济成本。我们提出批量提示（batch prompting）方法，这是一种简单而有效的提示策略，能使LLM以批量形式而非逐一样本进行推理。该方法在保持下游任务性能的同时，降低令牌（token）和时间成本。我们从理论上证明，在少样本上下文学习（few-shot in-context learning）设置下，推理成本几乎与每批样本数量成反比。我们通过在常识问答、算术推理以及自然语言推理/理解（NLI/NLU）等十个数据集上广泛验证了批量提示的有效性：批量提示显著降低了LLM（Codex）的推理令牌和时间成本（六样本批量时可降低高达5倍），同时取得了更优或相当的性能。对于最先进的基于聊天的LLM（如GPT-3.5和GPT-4），批量提示的优势同样存在。进一步分析表明，每批样本数量和任务复杂度会影响其性能。此外，批量提示可适用于基于LLM的不同推理方法。我们的代码见https://github.com/xlang-ai/batch-prompting。