The pervasiveness of proprietary language models has raised privacy concerns for users' sensitive data, emphasizing the need for private inference (PI), where inference is performed directly on encrypted inputs. However, current PI methods face prohibitively higher communication and latency overheads, primarily due to nonlinear operations. In this paper, we present a comprehensive analysis to understand the role of nonlinearities in transformer-based decoder-only language models. We introduce AERO, a four-step architectural optimization framework that refines the existing LLM architecture for efficient PI by systematically removing nonlinearities such as LayerNorm and GELU and reducing FLOPs counts. For the first time, we propose a Softmax-only architecture with significantly fewer FLOPs tailored for efficient PI. Furthermore, we devise a novel entropy regularization technique to improve the performance of Softmax-only models. AERO achieves up to 4.23$\times$ communication and 1.94$\times$ latency reduction. We validate the effectiveness of AERO by benchmarking it against the state-of-the-art.

翻译：专有语言模型的普及引发了用户敏感数据的隐私担忧，凸显了隐私推理（PI）的必要性，即在加密输入上直接执行推理。然而，当前的PI方法面临过高的通信和延迟开销，这主要源于非线性操作。本文中，我们提出了一项全面分析，以理解非线性操作在基于Transformer的解码器专用语言模型中的作用。我们介绍了AERO，一个四步架构优化框架，通过系统性地移除LayerNorm和GELU等非线性操作并减少浮点运算次数，对现有LLM架构进行改进以实现高效PI。我们首次提出了一种专为高效PI定制、浮点运算次数显著减少的纯Softmax架构。此外，我们设计了一种新颖的熵正则化技术以提升纯Softmax模型的性能。AERO实现了高达4.23倍的通信开销降低和1.94倍的延迟降低。我们通过与最先进技术进行基准测试，验证了AERO的有效性。