Managing data and code in open scientific research is complicated by two key problems: large datasets often cannot be stored alongside code in repository platforms like GitHub, and iterative analysis can lead to unnoticed changes to data, increasing the risk that analyses are based on older versions of data. Here, I introduce SciDataFlow: a fast, concurrent command-line tool paired with a simple Data Manifest specification. SciDataFlow streamlines tracking data changes, uploading data to remote repositories, and pulling in all data necessary to reproduce a computational analysis.
翻译:在开放科学研究中管理数据和代码面临两个关键问题:大型数据集通常无法与代码一同存储在GitHub等仓库平台中,且迭代分析可能导致数据变更未被察觉,从而增加分析基于旧版本数据的风险。本文介绍SciDataFlow:一款快速并发的命令行工具,配合简洁的数据清单规范。SciDataFlow可简化数据变更追踪、数据上传至远程仓库、以及获取复现计算分析所需全部数据的流程。