Recently, the study of linear misspecified bandits has generated intriguing implications of the hardness of learning in bandits and reinforcement learning (RL). In particular, Du et al. (2020) show that even if a learner is given linear features in $\mathbb{R}^d$ that approximate the rewards in a bandit or RL with a uniform error of $\varepsilon$, searching for an $O(\varepsilon)$-optimal action requires pulling at least $\Omega(\exp(d))$ queries. Furthermore, Lattimore et al. (2020) show that a degraded $O(\varepsilon\sqrt{d})$-optimal solution can be learned within $\operatorname{poly}(d/\varepsilon)$ queries. Yet it is unknown whether a structural assumption on the ground-truth parameter, such as sparsity, could break the $\varepsilon\sqrt{d}$ barrier. In this paper, we address this question by showing that algorithms can obtain $O(\varepsilon)$-optimal actions by querying $O(\varepsilon^{-s}d^s)$ actions, where $s$ is the sparsity parameter, removing the $\exp(d)$-dependence. We then establish information-theoretical lower bounds, i.e., $\Omega(\exp(s))$, to show that our upper bound on sample complexity is nearly tight if one demands an error $ O(s^{\delta}\varepsilon)$ for $0<\delta<1$. For $\delta\geq 1$, we further show that $\operatorname{poly}(s/\varepsilon)$ queries are possible when the linear features are "good" and even in general settings. These results provide a nearly complete picture of how sparsity can help in misspecified bandit learning and provide a deeper understanding of when linear features are "useful" for bandit and reinforcement learning with misspecification.

翻译：最近，对线性错误指定赌博机的研究揭示了赌博机和强化学习中学习难度的有趣含义。特别是，Du等人（2020）表明，即使学习者在$\mathbb{R}^d$中获得线性特征，这些特征能以$\varepsilon$的均匀误差近似赌博机或强化学习中的奖励，要找到一个$O(\varepsilon)$-最优动作至少需要$\Omega(\exp(d))$次查询。此外，Lattimore等人（2020）表明，可以在$\operatorname{poly}(d/\varepsilon)$次查询内学习到一个退化的$O(\varepsilon\sqrt{d})$-最优解。然而，尚不清楚对真实参数的结构性假设（如稀疏性）能否突破$\varepsilon\sqrt{d}$的障碍。在本文中，我们通过证明算法可以通过查询$O(\varepsilon^{-s}d^s)$个动作来获得$O(\varepsilon)$-最优动作来回答这个问题，其中$s$是稀疏参数，从而消除了$\exp(d)$依赖性。然后，我们建立了信息论下界，即$\Omega(\exp(s))$，以表明如果我们要求误差为$O(s^{\delta}\varepsilon)$，其中$0<\delta<1$，我们的样本复杂度上界几乎是紧的。对于$\delta\geq 1$，我们进一步表明，当线性特征“良好”时，甚至在一般情况下，$\operatorname{poly}(s/\varepsilon)$次查询是可能的。这些结果提供了关于稀疏性如何在错误指定赌博机学习中发挥作用的几乎完整的图景，并更深入地理解了线性特征何时对带有错误指定的赌博机和强化学习“有用”。