Dynamic treatment regimes (DTRs) formalize medical decision-making as a sequence of rules for different stages, mapping patient-level information to recommended treatments. In practice, estimating an optimal DTR using observational data from electronic medical record (EMR) databases can be complicated by covariates that are missing not at random (MNAR) due to informative monitoring of patients. Since complete case analysis can result in consistent estimation of outcome model parameters under the assumption of outcome-independent missingness, Q-learning is a natural approach to accommodating MNAR covariates. However, the backward induction algorithm used in Q-learning can introduce challenges, as MNAR covariates at later stages can result in MNAR pseudo-outcomes at earlier stages, leading to suboptimal DTRs, even if the longitudinal outcome variables are fully observed. To address this unique missing data problem in DTR settings, we propose two weighted Q-learning approaches where inverse probability weights for missingness of the pseudo-outcomes are obtained through estimating equations with valid nonresponse instrumental variables or sensitivity analysis. Asymptotic properties of the weighted Q-learning estimators are derived and the finite-sample performance of the proposed methods is evaluated and compared with alternative methods through extensive simulation studies. Using EMR data from the Medical Information Mart for Intensive Care database, we apply the proposed methods to investigate the optimal fluid strategy for sepsis patients in intensive care units.

翻译：动态治疗方案（DTRs）将医疗决策形式化为不同阶段的规则序列，将患者层面信息映射至推荐治疗方案。在实际应用中，利用电子病历（EMR）数据库的观察性数据估计最优DTR时，可能因患者信息性监测导致的非随机缺失（MNAR）协变量而变得复杂。由于在假设结局独立缺失条件下，完整案例分析可得到结局模型参数的一致估计，Q学习是处理MNAR协变量的自然方法。然而，Q学习使用的逆向归纳算法会引入新的挑战：后续阶段的MNAR协变量可能导致前期阶段的伪结局非随机缺失，即使纵向结局变量被完全观测，仍会生成次优DTR。为解决DTR设置中这一独特的缺失数据问题，我们提出两种加权Q学习方法，通过基于有效非应答工具变量或敏感性分析的估计方程获取伪结局缺失的逆概率权重。推导了加权Q学习估计量的渐近性质，并通过大量模拟研究评估和比较了所提方法的有限样本性能。利用重症监护医疗信息集市数据库的EMR数据，我们应用所提方法探究重症监护病房脓毒症患者的最优液体复苏策略。