An increasing number of reports raise concerns about the risk that machine learning algorithms could amplify health disparities due to biases embedded in the training data. Seyyed-Kalantari et al. find that models trained on three chest X-ray datasets yield disparities in false-positive rates (FPR) across subgroups on the 'no-finding' label (indicating the absence of disease). The models consistently yield higher FPR on subgroups known to be historically underserved, and the study concludes that the models exhibit and potentially even amplify systematic underdiagnosis. We argue that the experimental setup in the study is insufficient to study algorithmic underdiagnosis. In the absence of specific knowledge (or assumptions) about the extent and nature of the dataset bias, it is difficult to investigate model bias. Importantly, their use of test data exhibiting the same bias as the training data (due to random splitting) severely complicates the interpretation of the reported disparities.

翻译：越来越多的报告指出，机器学习算法可能因训练数据中的偏差而加剧健康差距，这引发了广泛担忧。Seyyed-Kalantari等人发现，基于三个胸部X光数据集训练的模型，在"无异常"标签（表示没有疾病）上，不同亚组间存在假阳性率差异。这些模型在历史上长期服务不足的亚组中持续表现出更高的假阳性率，研究据此得出结论，认为这些模型不仅反映了甚至可能加剧了系统性欠诊断。我们认为，该研究的实验设置不足以研究算法层面的欠诊断问题。在缺乏对数据集偏差程度和性质的具体认知（或假设）时，很难探究模型偏差。关键在于，他们采用与训练数据存在相同偏差的测试数据（源于随机分割），这严重干扰了对所报告差异的解读。