We introduce Coarse Q-learning (CQL), a reinforcement-learning model for bandit problems with stochastically varying menus. Alternatives are exogenously partitioned into similarity classes, and feedback from sampled alternatives is pooled within classes into class-level valuations. Choices follow multinomial logit over class valuations, and valuations update toward realized payoffs as in Q-learning. Using stochastic approximation, we derive the mean-field dynamics and characterize the steady states as smooth analogues of Valuation Equilibria. The model yields novel long-run phenomena in the high payoff-sensitivity limit: depending on the environment, CQL may exhibit multiple stable strict equilibria, a unique globally stable mixed equilibrium with indifference across classes, or no stable equilibrium at all, with valuations and choice probabilities converging instead to a stable limit cycle. These outcomes are driven by coarse aggregation and do not arise in the standard alternative-level benchmark.

翻译：我们提出粗糙Q学习（Coarse Q-learning, CQL），一种针对具有随机变化菜单的赌博机问题的强化学习模型。备选方案被外生地划分为若干相似性类别，来自被采样备选方案的反馈在类别内汇总为类别级估值。选择遵循基于类别估值的多项逻辑分布，估值通过类Q学习机制向已实现收益更新。利用随机逼近方法，我们推导出平均场动力学，并将稳态表征为估值均衡的平滑类比。该模型在高收益敏感性极限下呈现出新颖的长期现象：取决于环境，CQL可能表现出多个稳定的严格均衡、一个唯一全局稳定的混合均衡（类别间无差异），或根本无稳定均衡，此时估值和选择概率收敛至稳定的极限环。这些结果由粗糙聚合驱动，在标准备选方案级基准中不会出现。