We consider the problem of controlling an unknown stochastic linear system with quadratic costs - called the adaptive LQ control problem. We re-examine an approach called ''Reward Biased Maximum Likelihood Estimate'' (RBMLE) that was proposed more than forty years ago, and which predates the ''Upper Confidence Bound'' (UCB) method as well as the definition of ''regret'' for bandit problems. It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation. We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB algorithm that combines the penalty of the RBMLE method with the constraints of the UCB method, uniting the two approaches to optimism in the face of uncertainty. We establish that theoretically, this method retains $\Tilde{\mathcal{O}}(\sqrt{T})$ regret, the best-known so far. We further compare the empirical performance of the proposed Augmented RBMLE-UCB and the standard RBMLE (without the augmentation) with UCB, Thompson Sampling, Input Perturbation, Randomized Certainty Equivalence and StabL on many real-world examples including flight control of Boeing 747 and Unmanned Aerial Vehicle. We perform extensive simulation studies showing that the Augmented RBMLE consistently outperforms UCB, Thompson Sampling and StabL by a huge margin, while it is marginally better than Input Perturbation and moderately better than Randomized Certainty Equivalence.

翻译：我们考虑未知随机线性系统在二次成本下的控制问题——即自适应LQ控制问题。我们重新审视了一种名为"奖励偏置最大似然估计"（RBMLE）的方法，该方法提出于四十余年前，早于"上限置信区间"（UCB）方法以及针对赌博机问题中"遗憾"的定义。其核心思路是在参数估计准则中添加一项偏好更大奖励参数的正则项。我们揭示了RBMLE与UCB方法之间的内在联系，并据此提出一种增强型RBMLE-UCB算法——该算法融合了RBMLE方法的惩罚项与UCB方法的约束条件，将两种面对不确定性时的乐观主义策略统一起来。我们从理论上证明，该方法保留了$\Tilde{\mathcal{O}}(\sqrt{T})$的最优遗憾界。此外，我们在包括波音747飞行控制与无人机在内的多个实际案例中，将所提增强型RBMLE-UCB算法与标准RBMLE（未增强版本）、UCB、汤普森采样、输入扰动、随机化确定等价以及StabL方法进行了经验性能对比。大量仿真研究表明，增强型RBMLE算法的性能显著优于UCB、汤普森采样和StabL，略优于输入扰动，且适度优于随机化确定等价方法。