Replacing non-polynomial functions (e.g., non-linear activation functions such as ReLU) in a neural network with their polynomial approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called polynomial approximation of neural network (PANN) in this paper, is compatible with advanced cryptosystems to enable privacy-preserving model inference. Using ``highly precise'' approximation, state-of-the-art PANN offers similar inference accuracy as the underlying backbone model. However, little is known about the effect of approximation, and existing literature often determined the required approximation precision empirically. In this paper, we initiate the investigation of PANN as a standalone object. Specifically, our contribution is two-fold. Firstly, we provide an explanation on the effect of approximate error in PANN. In particular, we discovered that (1) PANN is susceptible to some type of perturbations; and (2) weight regularisation significantly reduces PANN's accuracy. We support our explanation with experiments. Secondly, based on the insights from our investigations, we propose solutions to increase inference accuracy for PANN. Experiments showed that combination of our solutions is very effective: at the same precision, our PANN is 10% to 50% more accurate than state-of-the-arts; and at the same accuracy, our PANN only requires a precision of $2^{-9}$ while state-of-the-art solution requires a precision of $2^{-12}$ using the ResNet-20 model on CIFAR-10 dataset.

翻译：用多项式近似替代神经网络中的非多项式函数（如ReLU等非线性激活函数）是隐私保护机器学习中的标准实践。由此产生的神经网络——本文称之为多项式近似神经网络（PANN）——可与高级密码系统兼容，从而实现隐私保护模型推理。通过使用"高精度"近似，最先进的PANN能获得与原始骨干模型相近的推理准确率。然而，关于近似效果的影响鲜有研究，现有文献通常凭经验确定所需的近似精度。本文首次将PANN作为独立对象展开研究，具体贡献体现在两个方面：首先，我们解释了PANN中近似误差的影响机制，特别发现：(1) PANN对某些类型的扰动敏感；(2) 权重正则化会显著降低PANN的准确率。我们通过实验支持这一解释。其次，基于研究中的发现，我们提出了提升PANN推理准确率的解决方案。实验表明，我们的方案组合效果显著：在相同精度下，我们的PANN比最先进方案准确率提升10%至50%；在相同准确率下，使用CIFAR-10数据集上的ResNet-20模型时，我们的PANN仅需$2^{-9}$的精度，而最先进方案需要$2^{-12}$的精度。