Deep learning models have shown promising predictive accuracy for time-series healthcare applications. However, ensuring the robustness of these models is vital for building trustworthy AI systems. Existing research predominantly focuses on robustness to synthetic adversarial examples, crafted by adding imperceptible perturbations to clean input data. However, these synthetic adversarial examples do not accurately reflect the most challenging real-world scenarios, especially in the context of healthcare data. Consequently, robustness to synthetic adversarial examples may not necessarily translate to robustness against naturally occurring adversarial examples, which is highly desirable for trustworthy AI. We propose a method to curate datasets comprised of natural adversarial examples to evaluate model robustness. The method relies on probabilistic labels obtained from automated weakly-supervised labeling that combines noisy and cheap-to-obtain labeling heuristics. Based on these labels, our method adversarially orders the input data and uses this ordering to construct a sequence of increasingly adversarial datasets. Our evaluation on six medical case studies and three non-medical case studies demonstrates the efficacy and statistical validity of our approach to generating naturally adversarial datasets

翻译：深度学习模型在时间序列医疗应用中展示了显著的预测准确性。然而，确保这些模型的鲁棒性对于构建可信赖的人工智能系统至关重要。现有研究主要关注对合成对抗样本的鲁棒性，这些样本通过向干净输入数据添加难以察觉的扰动生成。然而，这些合成对抗样本无法准确反映最具挑战性的现实场景，尤其是在医疗数据背景下。因此，对合成对抗样本的鲁棒性未必能转化为对自然出现的对抗样本的鲁棒性——而这正是可信赖人工智能所高度期望的。我们提出了一种方法，通过构建由自然对抗样本组成的数据集来评估模型鲁棒性。该方法依赖于从自动弱监督标注中获得的概率标签，该标注结合了噪声大且成本低廉的标注启发式规则。基于这些标签，我们的方法以对抗方式对输入数据进行排序，并利用这一顺序构建一系列对抗性逐步增强的数据集。我们在六个医疗案例研究和三个非医疗案例研究中的评估表明，该方法在生成自然对抗数据集方面具有有效性和统计有效性。