The reliability of machine learning critically depends on dataset quality. While machine learning applied to computer vision and natural language processing benefits from high-quality benchmark datasets, cyber security often falls behind, as quality ties to the ability of accessing hard-to-obtain realistic data that may evolve over time. Android is, however, positioned uniquely in this ecosystem due to AndroZoo and other sources, which provide large-scale, continuously updated, and timestamped repositories of benign and malicious apps. Since their release, such data sources provided access to populations of Android apps that researchers can sample from to evaluate learning-based methods in realistic settings, i.e., over temporal frames to account for app evolution (natural distribution shift) and test datasets that reflect in-the-wild class ratios. Surprisingly, we observe that despite this abundance of data, performance discrepancies of learning-based Android malware detectors still persist even after satisfying such realistic requirements, which challenges our ability to understand what the state of the art in this field is. In this work, we identify five novel factors that influence such discrepancies: we show how such factors have been largely overlooked and the impact they have on providing sound evaluations. Our findings and recommendations help define a methodology for curating trustworthy datasets towards sound evaluations of Android malware classifiers.

翻译：机器学习的可靠性关键取决于数据集质量。尽管应用于计算机视觉和自然语言处理的机器学习受益于高质量基准数据集，但网络安全领域往往落后，因为其质量与获取可能随时间演变的、难以获取的真实数据的能力密切相关。然而，由于AndroZoo及其他数据源提供了大规模、持续更新且带时间戳的良性及恶意应用仓库，安卓系统在该生态系统中占据独特地位。自这些数据源发布以来，研究人员得以获取安卓应用总体样本，从而在真实场景下评估基于学习的方法，即通过时间窗口来考虑应用演变（自然分布漂移），并使用反映野外类别比例的测试数据集。令人惊讶的是，我们观察到尽管数据充足，即使在满足此类真实需求后，基于学习的安卓恶意软件检测器的性能差异依然存在，这挑战了我们对该领域当前技术水平理解的能力。在本工作中，我们识别出影响此类差异的五项新因素：我们展示了这些因素如何被广泛忽视，以及它们对提供可靠评估产生的影响。我们的发现与建议有助于定义一种方法论，用于构建可信数据集以实现对安卓恶意软件分类器的可靠评估。