The recent advancements in deep learning have brought about significant changes in various aspects of people's lives. Meanwhile, these rapid developments have raised concerns about the legitimacy of the training process of deep neural networks. To protect the intellectual properties of AI developers, directly examining the training process by accessing the model parameters and training data is often prohibited for verifiers. In response to this challenge, we present zero-knowledge deep learning (zkDL), an efficient zero-knowledge proof for deep learning training. To address the long-standing challenge of verifiable computations of non-linearities in deep learning training, we introduce zkReLU, a specialized proof for the ReLU activation and its backpropagation. zkReLU turns the disadvantage of non-arithmetic relations into an advantage, leading to the creation of FAC4DNN, our specialized arithmetic circuit design for modelling neural networks. This design aggregates the proofs over different layers and training steps, without being constrained by their sequential order in the training process. With our new CUDA implementation that achieves full compatibility with the tensor structures and the aggregated proof design, zkDL enables the generation of complete and sound proofs in less than a second per batch update for an 8-layer neural network with 10M parameters and a batch size of 64, while provably ensuring the privacy of data and model parameters. To our best knowledge, we are not aware of any existing work on zero-knowledge proof of deep learning training that is scalable to million-size networks.

翻译：近期深度学习领域的进展显著改变了人们生活的方方面面，但随之而来的快速发展引发了人们对深度神经网络训练过程合法性的担忧。为保护AI开发者的知识产权，验证者通常被禁止通过访问模型参数和训练数据来直接检查训练过程。针对这一挑战，我们提出零知识深度学习（zkDL），一种针对深度学习训练的高效零知识证明系统。为解决深度学习训练中非线性运算可验证性这一长期难题，我们引入zkReLU，一种专门针对ReLU激活函数及其反向传播的证明方法。zkReLU将非算术关系的劣势转化为优势，从而构建了FAC4DNN——专用于神经网络建模的算术电路设计。该设计可实现不同层间及训练步骤间证明的聚合，且不受训练过程中顺序约束的限制。通过我们新实现的CUDA系统（完全兼容张量结构与聚合证明设计），zkDL能够为拥有1000万参数、批量大小为64的8层神经网络，在每次批次更新中生成完整且可靠的证明（耗时<1秒），同时可证明地保障数据与模型参数的隐私性。据我们所知，目前尚无其他关于深度学习训练零知识证明的研究可扩展至百万级参数规模的网络。