$\textbf{Objective:}$ High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, Numerous weakly-supervised learning methods have been proposed. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (e.g., sensitivity, specificity). $\textbf{Materials and Methods:}$ ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC through in-depth simulation studies and an extensive evaluation of six PAs from Mass General Brigham. $\textbf{Results:}$ In both simulated and real data, ssROC produced ROC parameter estimates with significantly lower variance than supROC for a fixed amount of labeled data. For the six PAs, the estimates from ssROC are approximately 40% less variable than supROC on average. $\textbf{Discussion:}$ ssROC enables precise evaluation of PA performance without demanding large volumes of labeled data. ssROC is also easily implementable in open-source $\texttt{R}$ software. $\textbf{Conclusion:}$ When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.

翻译：$\textbf{目的：}$ 高通量表型分析将加速电子健康记录在转化研究中的应用。当前的关键障碍在于表型算法估算与评估需要大量医学监督。为应对这一挑战，研究者已提出多种弱监督学习方法。然而，当仅有极小比例数据被标注时，可靠评估表型算法预测性能的方法仍然匮乏。为弥补这一空白，我们提出一种半监督方法（ssROC），用于估算表型算法接收者操作特征曲线参数（如灵敏度、特异度）。$\textbf{材料与方法：}$ ssROC利用少量标注数据集通过非参数方法对缺失标签进行插补，继而基于插补结果进行ROC参数估计，相较于仅使用标注数据的经典监督ROC分析，可获得更精确的表型算法性能估计。通过深度仿真研究及对马萨诸塞总医院布里格姆分院六种表型算法的系统评估，我们对ssROC进行了验证。$\textbf{结果：}$ 在仿真与真实数据中，对于固定标注数据量，ssROC生成的ROC参数估计值方差显著低于监督ROC分析。针对六种表型算法，ssROC估计值的变异性平均比监督ROC分析低约40%。$\textbf{讨论：}$ ssROC可在无需大量标注数据的情况下实现表型算法性能的精确评估，且易于在开源$\texttt{R}$软件中实现。$\textbf{结论：}$ 当与弱监督表型算法联合使用时，ssROC能为基于电子健康记录的研究提供可靠、高效的表型分析。