We consider the question of how to employ next-token prediction algorithms in adversarial online decision-making environments. Specifically, if we train a next-token prediction model on a distribution $\mathcal{D}$ over sequences of opponent actions, when is it the case that the induced online decision-making algorithm (by approximately best responding to the model's predictions) has low adversarial regret (i.e., when is $\mathcal{D}$ a \emph{low-regret distribution})? For unbounded context windows (where the prediction made by the model can depend on all the actions taken by the adversary thus far), we show that although not every distribution $\mathcal{D}$ is a low-regret distribution, every distribution $\mathcal{D}$ is exponentially close (in TV distance) to one low-regret distribution, and hence sublinear regret can always be achieved at negligible cost to the accuracy of the original next-token prediction model. In contrast to this, for bounded context windows (where the prediction made by the model can depend only on the past $w$ actions taken by the adversary, as may be the case in modern transformer architectures), we show that there are some distributions $\mathcal{D}$ of opponent play that are $Θ(1)$-far from any low-regret distribution $\mathcal{D'}$ (even when $w = Ω(T)$ and such distributions exist). Finally, we complement these results by showing that the unbounded context robustification procedure can be implemented by layers of a standard transformer architecture, and provide empirical evidence that transformer models can be efficiently trained to represent these new low-regret distributions.

翻译：我们探讨如何在对抗性在线决策环境中利用下一个令牌预测算法。具体而言，若我们在对手动作序列的分布 $\mathcal{D}$ 上训练一个下一个令牌预测模型，那么由该模型预测（通过对模型预测进行近似最优响应）所导出的在线决策算法何时具有低对抗性遗憾（即，$\mathcal{D}$ 何时是低遗憾分布）？对于无限上下文窗口（模型预测可依赖于对手迄今为止的所有动作），我们证明尽管并非所有分布 $\mathcal{D}$ 都是低遗憾分布，但每个分布 $\mathcal{D}$ 在总变差距离上指数接近某个低遗憾分布，因此始终能以原始下一个令牌预测模型精度的可忽略代价实现次线性遗憾。与此相反，对于有限上下文窗口（模型预测仅依赖于对手过去 $w$ 个动作，这可能是现代Transformer架构中的情况），我们证明存在某些对手对弈分布 $\mathcal{D}$ 与任何低遗憾分布 $\mathcal{D'}$ 相距 $\Theta(1)$（即使当 $w = \Omega(T)$ 时，此类分布依然存在）。最后，我们通过证明无限上下文鲁棒化过程可由标准Transformer架构的层实现来补充以上结果，并提供实验证据表明Transformer模型可被高效训练以表征这些新型低遗憾分布。