Surveys are commonly used to facilitate research in epidemiology, health, and the social and behavioral sciences. Often, these surveys are not simple random samples, and respondents are given weights reflecting their probability of selection into the survey. It is well known that analysts can use these survey weights to produce unbiased estimates of population quantities like totals. In this article, we show that survey weights also can be beneficial for evaluating the quality of predictive models when splitting data into training and test sets. In particular, we characterize model assessment statistics, such as sensitivity and specificity, as finite population quantities, and compute survey-weighted estimates of these quantities with sample test data comprising a random subset of the original data.Using simulations with data from the National Survey on Drug Use and Health and the National Comorbidity Survey, we show that unweighted metrics estimated with sample test data can misrepresent population performance, but weighted metrics appropriately adjust for the complex sampling design. We also show that this conclusion holds for models trained using upsampling for mitigating class imbalance. The results suggest that weighted metrics should be used when evaluating performance on sample test data.

翻译：调查通常用于促进流行病学、健康科学以及社会与行为科学的研究。这些调查往往并非简单随机样本，受访者会被赋予反映其入选调查概率的权重。众所周知，研究人员可利用这些调查权重生成总体数量（如总数）的无偏估计。本文证明，在将数据划分为训练集和测试集时，调查权重同样有助于评估预测模型的质量。具体而言，我们将敏感度和特异度等模型评估统计量定义为有限总体参数，并利用包含原始数据随机子集的样本测试数据计算这些参数的调查加权估计。通过基于全国药物使用与健康调查及全国共病调查数据的模拟实验，我们表明：使用样本测试数据估计的无加权指标可能歪曲总体表现，而加权指标能够针对复杂抽样设计进行适当调整。研究同时证明，即使对于为缓解类别不平衡采用上采样法训练的模型，该结论依然成立。结果建议，在评估样本测试数据的表现时应采用加权指标。