Research in various fields is currently experiencing challenges regarding the reproducibility of results. This problem is also prevalent in machine learning (ML) research. The issue arises, for example, due to unpublished data and/or source code and the sensitivity of ML training conditions. Although different solutions have been proposed to address this issue, such as using ML platforms, the level of reproducibility in ML-driven research remains unsatisfactory. Therefore, in this article, we discuss the reproducibility of ML-driven research with three main aims: (i) identifying the barriers to reproducibility when applying ML in research as well as categorize the barriers to different types of reproducibility (description, code, data, and experiment reproducibility), (ii) discussing potential drivers such as tools, practices, and interventions that support ML reproducibility, as well as distinguish between technology-driven drivers, procedural drivers, and drivers related to awareness and education, and (iii) mapping the drivers to the barriers. With this work, we hope to provide insights and to contribute to the decision-making process regarding the adoption of different solutions to support ML reproducibility.
翻译:当前,多个领域的研究在结果可复现性方面面临挑战。这一问题在机器学习(ML)研究中也普遍存在。例如,未公开的数据和/或源代码以及ML训练条件的敏感性导致了该问题的产生。尽管已有多种解决方案被提出以应对此问题,例如使用ML平台,但ML驱动的研究中的可复现性水平仍不尽如人意。因此,本文围绕ML驱动研究的可复现性展开讨论,主要目标有三:(i) 识别在研究应用ML时遇到的可复现性障碍,并将这些障碍归类至不同类型的可复现性(描述、代码、数据及实验可复现性);(ii) 探讨支持ML可复现性的潜在驱动因素,例如工具、实践与干预措施,并区分技术驱动因素、流程驱动因素以及与认知和教育相关的驱动因素;(iii) 将驱动因素与障碍进行映射关联。通过此项工作,我们期望为决策过程提供见解,助力于采纳不同解决方案以支持ML可复现性。