Run-by-run variability in parallel programs caused by floating-point non-associativity (FPNA) has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility negatively affects efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning (DL) training and inference pipelines to FPNA have been found to be extreme, and can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled DL models with high-performance computing (HPC) simulations, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of FPNA within modern parallel programming models, analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs, and examine the recently-added deterministic options within the PyTorch framework within the context of GPU deployment, uncovering and quantifying the impacts of input parameters triggering run-by-run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism provided by deterministic hardware, using the Groq LPU$^{TM}$ accelerator for inference portions of the DL pipeline. We demonstrate the benefits that this strategy can provide within reproducibility and correctness efforts.

翻译：浮点非结合性（FPNA）导致的并行程序运行间变异性，已知会因误差累积而显著影响迭代算法的可复现性。不可复现性对随机程序的正确性测试效率与效果产生负面影响。近期研究发现，深度学习（DL）训练与推理流程对FPNA的敏感性极为显著，可能阻碍商业应用的认证、鲁棒性与敏感性的准确评估以及缺陷检测。科学计算应用中的新方法将DL模型与高性能计算（HPC）模拟相结合，加剧了调试与测试的挑战。本文研究了现代并行编程模型中FPNA的统计特性，分析了在GPU上用确定性替代方案替换原子操作对性能与生产力的影响，并在GPU部署背景下检验了PyTorch框架新增的确定性选项，揭示了触发运行间变异性的输入参数影响并量化其效应，同时评估了相关文档的可靠性与完整性。最后，我们评估了利用确定性硬件提供的自动确定性策略，采用Groq LPU$^{TM}$加速器处理DL流程的推理部分，论证了该策略在提升可复现性与正确性方面的潜在优势。