In this work, we present a novel matrix-encoding method that is particularly convenient for neural networks to make predictions in a privacy-preserving manner using homomorphic encryption. Based on this encoding method, we implement a convolutional neural network for handwritten image classification over encryption. For two matrices $A$ and $B$ to perform homomorphic multiplication, the main idea behind it, in a simple version, is to encrypt matrix $A$ and the transpose of matrix $B$ into two ciphertexts respectively. With additional operations, the homomorphic matrix multiplication can be calculated over encrypted matrices efficiently. For the convolution operation, we in advance span each convolution kernel to a matrix space of the same size as the input image so as to generate several ciphertexts, each of which is later used together with the ciphertext encrypting input images for calculating some of the final convolution results. We accumulate all these intermediate results and thus complete the convolution operation. In a public cloud with 40 vCPUs, our convolutional neural network implementation on the MNIST testing dataset takes $\sim$ 287 seconds to compute ten likelihoods of 32 encrypted images of size $28 \times 28$ simultaneously. The data owner only needs to upload one ciphertext ($\sim 19.8$ MB) encrypting these 32 images to the public cloud.

翻译：本文提出了一种新颖的矩阵编码方法，该方法特别适用于利用同态加密以隐私保护方式让神经网络进行预测。基于该编码方法，我们实现了一个用于加密手写图像分类的卷积神经网络。为对两个矩阵 $A$ 和 $B$ 执行同态乘法，其核心思想（简化版本）是将矩阵 $A$ 和矩阵 $B$ 的转置分别加密为两个密文。通过附加操作，可在加密矩阵上高效计算同态矩阵乘法。针对卷积操作，我们预先将每个卷积核扩展至与输入图像相同大小的矩阵空间，从而生成多个密文，每个密文随后与加密输入图像的密文联合使用，以计算部分最终卷积结果。我们累加所有这些中间结果，从而完成卷积操作。在配备40个vCPU的公有云上，基于MNIST测试数据集的卷积神经网络实现仅需约287秒即可同时计算32幅大小为$28 \times 28$的加密图像的十个似然值。数据所有者只需向公有云上传一个加密这些32幅图像的密文（约19.8 MB）。