In this paper, we present a practical solution to implement privacy-preserving CNN training based on mere Homomorphic Encryption (HE) technique. To our best knowledge, this is the first attempt successfully to crack this nut and no work ever before has achieved this goal. Several techniques combine to accomplish the task:: (1) with transfer learning, privacy-preserving CNN training can be reduced to homomorphic neural network training, or even multiclass logistic regression (MLR) training; (2) via a faster gradient variant called $\texttt{Quadratic Gradient}$, an enhanced gradient method for MLR with a state-of-the-art performance in convergence speed is applied in this work to achieve high performance; (3) we employ the thought of transformation in mathematics to transform approximating Softmax function in the encryption domain to the approximation of the Sigmoid function. A new type of loss function termed $\texttt{Squared Likelihood Error}$ has been developed alongside to align with this change.; and (4) we use a simple but flexible matrix-encoding method named $\texttt{Volley Revolver}$ to manage the data flow in the ciphertexts, which is the key factor to complete the whole homomorphic CNN training. The complete, runnable C++ code to implement our work can be found at: \href{https://github.com/petitioner/HE.CNNtraining}{$\texttt{https://github.com/petitioner/HE.CNNtraining}$}. We select $\texttt{REGNET\_X\_400MF}$ as our pre-trained model for transfer learning. We use the first 128 MNIST training images as training data and the whole MNIST testing dataset as the testing data. The client only needs to upload 6 ciphertexts to the cloud and it takes $\sim 21$ mins to perform 2 iterations on a cloud with 64 vCPUs, resulting in a precision of $21.49\%$.

翻译：本文提出了一种基于纯同态加密（HE）技术实现保护隐私的CNN训练的实用方案。据我们所知，这是首次成功攻克这一难题的研究，此前尚无任何工作实现这一目标。多种技术协同完成该任务：（1）通过迁移学习，保护隐私的CNN训练可简化为同态神经网络训练，甚至多类逻辑回归（MLR）训练；（2）采用名为$\texttt{Quadratic Gradient}$的快速梯度变体，该增强型梯度方法在MLR收敛速度方面达到当前最优性能，进而实现高性能；（3）运用数学中的变换思想，将加密域中逼近Softmax函数的问题转化为对Sigmoid函数的逼近问题。为此同步开发了新型损失函数$\texttt{平方似然误差}$（$\texttt{Squared Likelihood Error}$）以适应这一变化；（4）采用名为$\texttt{Volley Revolver}$的简单灵活矩阵编码方法来管理密文数据流，这是完成整个同态CNN训练的关键因素。实现本工作的完整可运行C++代码可见于：\href{https://github.com/petitioner/HE.CNNtraining}{$\texttt{https://github.com/petitioner/HE.CNNtraining}$}。我们选择$\texttt{REGNET\_X\_400MF}$作为迁移学习的预训练模型，使用MNIST前128张训练图像作为训练数据，完整MNIST测试集作为测试数据。客户端仅需上传6个密文至云端，在配备64个vCPU的云服务器上执行2次迭代需时约21分钟，最终获得$21.49\%$的精确度。