The increasing compute demands of AI systems has led to the emergence of services that train models on behalf of clients lacking necessary resources. However, ensuring correctness of training and guarding against potential training-time attacks, such as data poisoning, poses challenges. Existing works on verifiable training largely fall into two classes: proof-based systems, which struggle to scale due to requiring cryptographic techniques, and "optimistic" methods that consider a trusted third-party auditor who replicates the training process. A key challenge with the latter is that hardware nondeterminism between GPU types during training prevents an auditor from replicating the training process exactly, and such schemes are therefore non-robust. We propose a method that combines training in a higher precision than the target model, rounding after intermediate computation steps, and storing rounding decisions based on an adaptive thresholding procedure, to successfully control for nondeterminism. Across three different NVIDIA GPUs (A40, Titan XP, RTX 2080 Ti), we achieve exact training replication at FP32 precision for both full-training and fine-tuning of ResNet-50 (23M) and GPT-2 (117M) models. Our verifiable training scheme significantly decreases the storage and time costs compared to proof-based systems.

翻译：人工智能系统日益增长的计算需求催生了代表缺乏必要资源的客户训练模型的服务。然而，确保训练的正确性并防范潜在的训练时攻击（如数据投毒）仍面临挑战。现有验证训练研究主要分为两类：基于证明的系统（因依赖密码学技术而难以扩展）和“乐观”方法（考虑可信第三方审核者复制训练过程）。后者面临的关键挑战在于，训练期间不同GPU类型之间的硬件非确定性会导致审核者无法精确复制训练过程，因此这类方案缺乏鲁棒性。我们提出一种方法，通过以比目标模型更高的精度进行训练、对中间计算步骤进行舍入，并基于自适应阈值程序存储舍入决策，成功实现了对非确定性的控制。在三种不同NVIDIA GPU（A40、Titan XP、RTX 2080 Ti）上，我们实现了ResNet-50（23M）和GPT-2（117M）模型在全训练与微调中FP32精度的精确训练复制。与基于证明的系统相比，我们的验证训练方案显著降低了存储与时间成本。