The increasing compute demands of AI systems has led to the emergence of services that train models on behalf of clients lacking necessary resources. However, ensuring correctness of training and guarding against potential training-time attacks, such as data poisoning, poses challenges. Existing works on verifiable training largely fall into two classes: proof-based systems, which struggle to scale due to requiring cryptographic techniques, and "optimistic" methods that consider a trusted third-party auditor who replicates the training process. A key challenge with the latter is that hardware nondeterminism between GPU types during training prevents an auditor from replicating the training process exactly, and such schemes are therefore non-robust. We propose a method that combines training in a higher precision than the target model, rounding after intermediate computation steps, and storing rounding decisions based on an adaptive thresholding procedure, to successfully control for nondeterminism. Across three different NVIDIA GPUs (A40, Titan XP, RTX 2080 Ti), we achieve exact training replication at FP32 precision for both full-training and fine-tuning of ResNet-50 (23M) and GPT-2 (117M) models. Our verifiable training scheme significantly decreases the storage and time costs compared to proof-based systems.

翻译：人工智能系统日益增长的计算需求催生了在缺乏必要资源的客户方代训练模型的服务。然而，确保训练正确性并防范潜在训练时攻击（如数据投毒）仍面临挑战。现有关于可验证训练的研究主要分为两类：基于证明的系统（因需要密码学技术而难以扩展）和"乐观"方法（依赖可信第三方审计员复制训练过程）。后者的核心挑战在于，不同GPU类型在训练过程中的硬件非确定性会阻碍审计员完全复制训练过程，导致此类方案缺乏鲁棒性。我们提出一种融合三项机制的方法：采用高于目标模型的精度进行训练、对中间计算步骤进行舍入操作，以及基于自适应阈值流程存储舍入决策，从而成功实现对非确定性的控制。在三种不同NVIDIA GPU（A40、Titan XP、RTX 2080 Ti）上，我们实现了ResNet-50（23M）和GPT-2（117M）模型在完整训练与微调场景下的FP32精度精确训练复制。与基于证明的系统相比，我们的可验证训练方案显著降低了存储与时间成本。