We introduce Coarse Q-learning (CQL), a reinforcement-learning model for bandit problems with stochastically varying menus. Alternatives are exogenously partitioned into similarity classes, and feedback from sampled alternatives is pooled within classes into class-level valuations. Choices follow multinomial logit over class valuations, and valuations update toward realized payoffs as in Q-learning. Using stochastic approximation, we derive the mean-field dynamics and characterize the steady states as smooth analogues of Valuation Equilibria. The model yields novel long-run phenomena in the high payoff-sensitivity limit: depending on the environment, CQL may exhibit multiple stable strict equilibria, a unique globally stable mixed equilibrium with indifference across classes, or no stable equilibrium at all, with valuations and choice probabilities converging instead to a stable limit cycle. These outcomes are driven by coarse aggregation and do not arise in the standard alternative-level benchmark.

翻译：我们提出粗Q学习（CQL），一种针对菜单随机变化的赌博机问题的强化学习模型。备选方案被外生地划分为相似性类别，采样备选方案的反馈在类别内聚合为类别级估值。选择遵循基于类别估值的多项Logit模型，估值像Q学习一样向实现收益更新。利用随机逼近方法，我们推导出平均场动力学，并将稳态刻画为估值均衡的光滑类比。模型在高收益敏感性极限下产生了新颖的长期现象：根据环境不同，CQL可能呈现多重稳定严格均衡、所有类别间漠然的唯一全局稳定混合均衡，或根本不存在稳定均衡——此时估值和选择概率收敛于稳定极限环。这些结果由粗粒化聚合驱动，在标准备选方案级基准中不会出现。