Major depressive disorder (MDD) presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD. To measure reward processing, patients perform computer-based behavioral tasks that involve making choices or responding to stimulants that are associated with different outcomes. Reinforcement learning (RL) models are fitted to extract parameters that measure various aspects of reward processing to characterize how patients make decisions in behavioral tasks. Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model; instead, there may be a switching of decision-making processes between multiple strategies. An important scientific question is how the dynamics of learning strategies in decision-making affect the reward learning ability of individuals with MDD. Motivated by the probabilistic reward task (PRT) within the EMBARC study, we propose a novel RL-HMM framework for analyzing reward-based decision-making. Our model accommodates learning strategy switching between two distinct approaches under a hidden Markov model (HMM): subjects making decisions based on the RL model or opting for random choices. We account for continuous RL state space and allow time-varying transition probabilities in the HMM. We introduce a computationally efficient EM algorithm for parameter estimation and employ a nonparametric bootstrap for inference. We apply our approach to the EMBARC study to show that MDD patients are less engaged in RL compared to the healthy controls, and engagement is associated with brain activities in the negative affect circuitry during an emotional conflict task.

翻译：重度抑郁症（MDD）因其复杂异质性在诊断和治疗方面面临挑战。新近证据表明，奖赏处理异常可能作为MDD的行为标志。为测量奖赏处理，患者需完成基于计算机的行为任务，包括对关联不同结果的刺激做出选择或反应。通过拟合强化学习（RL）模型提取参数，可量化奖赏处理的多个维度，进而表征患者在行为任务中的决策模式。最新研究发现，仅基于单一RL模型刻画奖赏学习存在局限性——决策过程可能在多种策略间切换。重要科学问题在于：决策过程中学习策略的动态变化如何影响MDD患者的奖赏学习能力？受EMBARC研究中概率性奖赏任务（PRT）启发，我们提出新颖的RL-HMM框架分析基于奖赏的决策。该模型在隐马尔可夫模型（HMM）框架下容纳两种学习策略的切换：被试依据RL模型决策或随机选择。我们考虑了连续RL状态空间，允许HMM中时变转移概率，并引入计算高效的EM算法进行参数估计，采用非参数自助法进行推断。将本方法应用于EMBARC研究表明：与健康对照组相比，MDD患者的RL参与度降低，且该参与度与情绪冲突任务中负性情感环路的大脑活动相关。