The primary way of building AI applications is shifting from training specialist models to prompting generalist models. A common practice for prompting generalist models, often referred to as in-context learning, is to append a few examples (demonstrations) to the prompt to help the model better understand the task. While effective, in-context learning can be inefficient because it makes the input prompt much longer, consuming valuable space in the context window and leading to larger computational costs. In this paper, we propose DynaICL, a recipe for efficient prompting with black-box generalist models that dynamically allocate in-context examples according to the input complexity and the computational budget. To achieve this, we train a meta controller that predicts the number of in-context examples suitable for the generalist model to make a good prediction based on the performance-efficiency trade-off for a specific input. We then dynamically allocate the number of demonstrations for an input according to predictions from the meta controller and the given computation budget. Experimental results show that dynamic example allocation helps achieve a better performance-efficiency trade-off in two practical settings where computational resources or the required performance is constrained. Specifically, DynaICL saves up to 46% token budget compared to the common practice that allocates the same number of in-context examples to each input. We also find that a meta controller trained on a certain backbone model and tasks can successfully generalize to unseen models and tasks.

翻译：构建AI应用的主要方式正从训练专用模型转向提示通用模型。提示通用模型的常见做法（常称为上下文学习）是在提示中附加若干示例（演示）以帮助模型更好地理解任务。虽然有效，但上下文学习可能效率低下，因为它使输入提示变长，占用上下文窗口的宝贵空间并导致更大的计算成本。本文提出DynaICL——一种针对黑箱通用模型的高效提示方法，可根据输入复杂度和计算预算动态分配上下文示例。为此，我们训练一个元控制器，基于特定输入的性能-效率权衡，预测通用模型进行准确预测所需的合适上下文示例数量。随后根据元控制器的预测和给定计算预算，动态分配每个输入的演示数量。实验结果表明，在计算资源或性能要求受限的两种实际场景中，动态示例分配有助于实现更优的性能-效率权衡。具体而言，与为每个输入分配相同数量上下文示例的常规做法相比，DynaICL最高可节省46%的令牌预算。我们还发现，基于特定骨干模型和任务训练的元控制器能够成功泛化到未见模型和任务上。