In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret. Like many other machine learning algorithms, the performance of bandits heavily depends on the values of hyperparameters, and theoretically derived parameter values may lead to unsatisfactory results in practice. Moreover, it is infeasible to use offline tuning methods like cross-validation to choose hyperparameters under the bandit environment, as the decisions should be made in real-time. To address this challenge, we propose the first online continuous hyperparameter tuning framework for contextual bandits to learn the optimal parameter configuration in practice within a search space on the fly. Specifically, we use a double-layer bandit framework named CDT (Continuous Dynamic Tuning) and formulate the hyperparameter optimization as a non-stationary continuum-armed bandit, where each arm represents a combination of hyperparameters, and the corresponding reward is the algorithmic result. For the top layer, we propose the Zooming TS algorithm that utilizes Thompson Sampling (TS) for exploration and a restart technique to get around the \textit{switching} environment. The proposed CDT framework can be easily utilized to tune contextual bandit algorithms without any pre-specified candidate set for multiple hyperparameters. We further show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.

翻译：在随机上下文赌博机中，智能体基于过往经验从随时间变化的动作集合中顺序选择动作，以最小化累积遗憾。与许多其他机器学习算法类似，赌博机的性能高度依赖超参数取值，而理论推导的参数值在实践中可能产生不理想的结果。此外，在赌博机环境下无法使用交叉验证等离线调优方法选择超参数，因为决策必须实时进行。针对这一挑战，我们提出首个上下文赌博机在线连续超参数调优框架，实现在搜索空间中实时学习最优参数配置。具体地，我们采用名为CDT（连续动态调优）的双层赌博机框架，将超参数优化建模为非平稳连续臂赌博机，其中每臂代表超参数组合，对应奖励为算法运行结果。在顶层，我们提出Zooming TS算法，该算法利用汤普森采样进行探索，并引入重启技术以应对\textit{切换}环境。提出的CDT框架可轻松用于调优上下文赌博机算法，无需为多个超参数预设候选集。我们进一步证明该框架理论上可实现次线性遗憾，并在合成数据集与真实数据集上持续优于所有现有方法。