Major depressive disorder (MDD) presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD. To measure reward processing, patients perform computer-based behavioral tasks that involve making choices or responding to stimulants that are associated with different outcomes. Reinforcement learning (RL) models are fitted to extract parameters that measure various aspects of reward processing to characterize how patients make decisions in behavioral tasks. Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model; instead, there may be a switching of decision-making processes between multiple strategies. An important scientific question is how the dynamics of learning strategies in decision-making affect the reward learning ability of individuals with MDD. Motivated by the probabilistic reward task (PRT) within the EMBARC study, we propose a novel RL-HMM framework for analyzing reward-based decision-making. Our model accommodates learning strategy switching between two distinct approaches under a hidden Markov model (HMM): subjects making decisions based on the RL model or opting for random choices. We account for continuous RL state space and allow time-varying transition probabilities in the HMM. We introduce a computationally efficient EM algorithm for parameter estimation and employ a nonparametric bootstrap for inference. We apply our approach to the EMBARC study to show that MDD patients are less engaged in RL compared to the healthy controls, and engagement is associated with brain activities in the negative affect circuitry during an emotional conflict task.

翻译：重度抑郁症（MDD）因其复杂性和异质性，在诊断和治疗方面均面临挑战。新出现的证据表明，奖赏处理异常可能作为MDD的一种行为标记。为测量奖赏处理，患者需执行基于计算机的行为任务，这些任务涉及做出选择或对与不同结果相关联的刺激物作出反应。通过拟合强化学习（RL）模型，可提取用于衡量奖赏处理各个方面的参数，以表征患者在行为任务中如何做出决策。近期研究发现，仅基于单一RL模型来表征奖赏学习存在不足；相反，决策过程可能在多种策略之间切换。一个重要的科学问题是：决策中学习策略的动态变化如何影响MDD个体的奖赏学习能力？受EMBARC研究中的概率奖赏任务（PRT）启发，我们提出了一种新颖的RL-HMM框架，用于分析基于奖赏的决策。我们的模型在隐马尔可夫模型（HMM）框架下，容纳了两种不同方法之间的学习策略切换：受试者基于RL模型做出决策或选择随机决策。我们考虑了连续的RL状态空间，并允许HMM中具有时变的转移概率。我们引入了一种计算高效的EM算法进行参数估计，并采用非参数自助法进行统计推断。我们将该方法应用于EMBARC研究，结果表明：与健康对照组相比，MDD患者较少采用RL策略进行决策，且这种参与度与情绪冲突任务期间负性情感神经回路的大脑活动相关。