Private computation of nonlinear functions, such as Rectified Linear Units (ReLUs) and max-pooling operations, in deep neural networks (DNNs) poses significant challenges in terms of storage, bandwidth, and time consumption. To address these challenges, there has been a growing interest in utilizing privacy-preserving techniques that leverage polynomial activation functions and kernelized convolutions as alternatives to traditional ReLUs. However, these alternative approaches often suffer from a trade-off between achieving faster private inference (PI) and sacrificing model accuracy. In particular, when applied to much deeper networks, these methods encounter training instabilities, leading to issues like exploding gradients (resulting in NaNs) or suboptimal approximations. In this study, we focus on PolyKervNets, a technique known for offering improved dynamic approximations in smaller networks but still facing instabilities in larger and more complex networks. Our primary objective is to empirically explore optimization-based training recipes to enhance the performance of PolyKervNets in larger networks. By doing so, we aim to potentially eliminate the need for traditional nonlinear activation functions, thereby advancing the state-of-the-art in privacy-preserving deep neural network architectures. Code can be found on GitHub at: \url{https://github.com/tolusophy/PolyKervNets/}

翻译：深度神经网络中非线性函数（如修正线性单元和最大池化操作）的私有计算在存储、带宽和时间消耗方面面临重大挑战。为解决这些问题，利用多项式激活函数和核化卷积作为传统ReLU替代方案的隐私保护技术日益受到关注。然而，这些替代方法往往需要在实现快速私有推理与牺牲模型精度之间进行权衡。尤其当应用于更深的网络时，这些方法会遭遇训练不稳定性，导致梯度爆炸（产生NaN值）或次优逼近等问题。本研究聚焦于PolyKervNets——一种在较小网络中能提供改进动态逼近但在更大更复杂网络中仍存在不稳定性的技术。我们的主要目标是实证探索基于优化的训练方法，以提升PolyKervNets在更大网络中的性能。通过此举，我们旨在潜在地消除对传统非线性激活函数的需求，从而推进隐私保护深度神经网络架构的技术前沿。代码已在GitHub上发布：\url{https://github.com/tolusophy/PolyKervNets/}