We study a $K$-armed bandit with delayed feedback and intermediate observations. We consider a model where intermediate observations have a form of a finite state, which is observed immediately after taking an action, whereas the loss is observed after an adversarially chosen delay. We show that the regime of the mapping of states to losses determines the complexity of the problem, irrespective of whether the mapping of actions to states is stochastic or adversarial. If the mapping of states to losses is adversarial, then the regret rate is of order $\sqrt{(K+d)T}$ (within log factors), where $T$ is the time horizon and $d$ is a fixed delay. This matches the regret rate of a $K$-armed bandit with delayed feedback and without intermediate observations, implying that intermediate observations are not helpful. However, if the mapping of states to losses is stochastic, we show that the regret grows at a rate of $\sqrt{\big(K+\min\{|\mathcal{S}|,d\}\big)T}$ (within log factors), implying that if the number $|\mathcal{S}|$ of states is smaller than the delay, then intermediate observations help. We also provide refined high-probability regret upper bounds for non-uniform delays, together with experimental validation of our algorithms.

翻译：我们研究一种具有延迟反馈和中间观察的$K$臂赌博机问题。考虑一个模型，其中中间观察以有限状态的形式出现，并在采取动作后立即被观察到，而损失则在对立选择的延迟之后被观察到。我们证明，无论动作到状态的映射是随机的还是对立的，状态到损失的映射机制决定了问题的复杂度。如果状态到损失的映射是对立的，那么遗憾率的阶为$\sqrt{(K+d)T}$（在对数因子内），其中$T$是时间范围，$d$是固定延迟。这与没有中间观察的延迟反馈$K$臂赌博机的遗憾率匹配，表明中间观察没有帮助。然而，如果状态到损失的映射是随机的，我们证明遗憾以$\sqrt{\big(K+\min\{|\mathcal{S}|,d\}\big)T}$（在对数因子内）的速率增长，这意味着如果状态数$|\mathcal{S}|$小于延迟，中间观察是有帮助的。我们还为非均匀延迟提供了精细的高概率遗憾上界，并附有对我们算法的实验验证。