$\textbf{Objective:}$ High-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed to estimate PAs. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (e.g., sensitivity, specificity). $\textbf{Materials and Methods:}$ ssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC through in-depth simulation studies and an extensive evaluation of eight PAs from Mass General Brigham. $\textbf{Results:}$ In both simulated and real data, ssROC produced ROC parameter estimates with significantly lower variance than supROC for a given amount of labeled data. For the eight PAs, our results illustrate that ssROC achieves similar precision to supROC, but with approximately 60% of the amount of labeled data on average. $\textbf{Discussion:}$ ssROC enables precise evaluation of PA performance to increase trust in observational health research without demanding large volumes of labeled data. ssROC is also easily implementable in open-source $\texttt{R}$ software. $\textbf{Conclusion:}$ When used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.

翻译：$\textbf{目的:}$ 高通量表型分析将加速电子健康记录(EHR)在转化研究中的应用。其中关键障碍在于表型算法(PA)的估计与评估需要大量医学监督。为应对这一挑战，研究者提出了众多弱监督学习方法用于估计PA。然而，在仅有极小比例数据被标注的情况下，尚缺乏可靠评估PA预测性能的方法。为填补这一空白，我们提出一种半监督方法(ssROC)，用于估计PA的受试者工作特征(ROC)参数（如灵敏度、特异度）。$\textbf{材料与方法:}$ ssROC利用小型标注数据集对缺失标签进行非参数插补，继而基于插补结果进行ROC参数估计，相较于仅使用标注数据的经典监督ROC分析(supROC)，可得到更精确的PA性能估计。通过深入模拟研究及对马萨诸塞总医院布列根分院八种PA的广泛评估，我们对ssROC进行了验证。$\textbf{结果:}$ 在模拟数据与真实数据中，对于给定数量的标注数据，ssROC产生的ROC参数估计方差显著低于supROC。针对八种PA的分析表明，ssROC可实现与supROC相当的精度，而平均仅需约60%的标注数据量。$\textbf{讨论:}$ ssROC无需大量标注数据即可实现PA性能的精确评估，从而提升观察性健康研究的可信度。同时，ssROC易于在开源$\texttt{R}$软件中实现。$\textbf{结论:}$ 当与弱监督PA联合使用时，ssROC能够促进基于EHR研究所需的可靠且高效的表型分析流程。