Dynamic treatment regimes (DTRs) formalize medical decision-making as a sequence of rules for different stages, mapping patient-level information to recommended treatments. In practice, estimating an optimal DTR using observational data from electronic medical record (EMR) databases can be complicated by covariates that are missing not at random (MNAR) due to informative monitoring of patients. Since complete case analysis can result in consistent estimation of outcome model parameters under the assumption of outcome-independent missingness \citep{Yang_Wang_Ding_2019}, Q-learning is a natural approach to accommodating MNAR covariates. However, the backward induction algorithm used in Q-learning can introduce complications, as MNAR covariates at later stages can result in MNAR pseudo-outcomes at earlier stages, leading to suboptimal DTRs, even if outcome variables are fully observed. To address this unique missing data problem in DTR settings, we propose two weighted Q-learning approaches where inverse probability weights for missingness of the pseudo-outcomes are obtained through estimating equations with valid nonresponse instrumental variables or sensitivity analysis. Asymptotic properties of the weighted Q-learning estimators are derived and the finite-sample performance of the proposed methods is evaluated and compared with alternative methods through extensive simulation studies. Using EMR data from the Medical Information Mart for Intensive Care database, we apply the proposed methods to investigate the optimal fluid strategy for sepsis patients in intensive care units.

翻译：动态治疗方案（DTR）将医疗决策形式化为不同阶段的规则序列，将患者层面信息映射至推荐治疗。在实际应用中，利用电子病历（EMR）数据库中的观察性数据估计最优DTR时，可能因患者信息性监测导致的非随机缺失（MNAR）协变量而变得复杂。由于在结果独立缺失假设下，完整病例分析可得到结果模型参数的一致估计（Yang et al., 2019），Q学习是处理MNAR协变量的自然方法。但Q学习中的反向归纳算法可能引入复杂问题：后期阶段的MNAR协变量会导致前期阶段产生非随机缺失的伪结果，即使在结果变量完全观测的情况下，也可能导致次优DTR。为解决DTR场景中这一独特的数据缺失问题，我们提出两种加权Q学习方法：通过利用有效无应答工具变量或敏感性分析的估计方程，获取伪结果缺失的逆概率权重。本文推导了加权Q学习估计量的渐近性质，并通过大规模仿真研究评估所提方法的有限样本性能，同时与替代方法进行比较。利用重症监护医学信息集市数据库中的EMR数据，我们将所提方法应用于重症监护病房脓毒症患者的最优液体复苏策略研究。