Bandit optimization usually refers to the class of online optimization problems with limited feedback, namely, a decision maker uses only the objective value at the current point to make a new decision and does not have access to the gradient of the objective function. While this name accurately captures the limitation in feedback, it is somehow misleading since it does not have any connection with the multi-armed bandits (MAB) problem class. We propose two new classes of problems: the functional multi-armed bandit problem (FMAB) and the best function identification problem. They are modifications of a multi-armed bandit problem and the best arm identification problem, respectively, where each arm represents an unknown black-box function. These problem classes are a surprisingly good fit for modeling real-world problems such as competitive LLM training. To solve the problems from these classes, we propose a new reduction scheme to construct UCB-type algorithms, namely, the F-LCB algorithm, based on algorithms for nonlinear optimization with known convergence rates. We provide the regret upper bounds for this reduction scheme based on the base algorithms' convergence rates. We add numerical experiments that demonstrate the performance of the proposed scheme.

翻译：老虎机优化通常指一类反馈受限的在线优化问题：决策者仅能依据当前点的目标函数值进行新决策，且无法获取目标函数的梯度信息。尽管这一名称准确反映了反馈的局限性，却存在一定误导性，因其与多臂老虎机问题类别并无实质关联。本文提出两个新的问题类别：函数式多臂老虎机问题与最优函数辨识问题。它们分别是多臂老虎机问题与最优臂辨识问题的扩展形式，其中每个臂代表一个未知的黑箱函数。这些新问题类别能出色地建模现实场景（例如竞争性大语言模型训练）。针对这两类问题，我们提出一种基于非线性优化算法（具有已知收敛速率）构建上置信界类算法的新规约框架——即F-LCB算法。我们依据基础算法的收敛速率，给出了该规约框架的遗憾上界。数值实验进一步验证了所提框架的性能表现。