Designing privacy-preserving deep learning models is a major challenge within the deep learning community. Homomorphic Encryption (HE) has emerged as one of the most promising approaches in this realm, enabling the decoupling of knowledge between the model owner and the data owner. Despite extensive research and application of this technology, primarily in convolutional neural networks, incorporating HE into transformer models has been challenging because of the difficulties in converting these models into a polynomial form. We break new ground by introducing the first polynomial transformer, providing the first demonstration of secure inference over HE with transformers. This includes a transformer architecture tailored for HE, alongside a novel method for converting operators to their polynomial equivalent. This innovation enables us to perform secure inference on LMs with WikiText-103. It also allows us to perform image classification with CIFAR-100 and Tiny-ImageNet. Our models yield results comparable to traditional methods, bridging the performance gap with transformers of similar scale and underscoring the viability of HE for state-of-the-art applications. Finally, we assess the stability of our models and conduct a series of ablations to quantify the contribution of each model component.

翻译：设计隐私保护的深度学习模型是深度学习领域的一项重大挑战。同态加密（HE）已成为该领域最有前景的方法之一，它能够实现模型所有者与数据所有者之间的知识解耦。尽管该技术已在卷积神经网络中得到广泛研究和应用，但由于将Transformer模型转换为多项式形式存在困难，将HE融入Transformer模型一直具有挑战性。我们率先提出首个多项式Transformer，首次展示了基于HE的Transformer安全推理。这包括一种专为HE设计的Transformer架构，以及一种将算子转换为其多项式等价形式的新方法。这一创新使我们能够在WikiText-103上对语言模型进行安全推理，同时也支持在CIFAR-100和Tiny-ImageNet上执行图像分类。我们的模型取得了与传统方法相当的结果，缩小了与类似规模Transformer的性能差距，凸显了HE在尖端应用中的可行性。最后，我们评估了模型的稳定性，并进行了一系列消融实验，以量化每个模型组件的贡献。