Convolutional neural network (CNN) inference using fully homomorphic encryption (FHE) is a promising private inference (PI) solution due to the capability of FHE that enables offloading the whole computation process to the server while protecting the privacy of sensitive user data. Prior FHE-based CNN (HCNN) work has demonstrated the feasibility of constructing deep neural network architectures such as ResNet using FHE. Despite these advancements, HCNN still faces significant challenges in practicality due to the high computational and memory overhead. To overcome these limitations, we present HyPHEN, a deep HCNN construction that incorporates novel convolution algorithms (RAConv and CAConv), data packing methods (2D gap packing and PRCR scheme), and optimization techniques tailored to HCNN construction. Such enhancements enable HyPHEN to substantially reduce the memory footprint and the number of expensive homomorphic operations, such as ciphertext rotation and bootstrapping. As a result, HyPHEN brings the latency of HCNN CIFAR-10 inference down to a practical level at 1.4 seconds (ResNet-20) and demonstrates HCNN ImageNet inference for the first time at 14.7 seconds (ResNet-18).

翻译：基于全同态加密（FHE）的卷积神经网络（CNN）推理是一种有前景的隐私推理（PI）解决方案，其优势在于FHE能够将整个计算过程外包给服务器，同时保护敏感用户数据的隐私性。现有基于FHE的CNN（HCNN）研究已证明，利用FHE构建深度神经网络架构（如ResNet）的可行性。尽管取得了这些进展，HCNN仍面临计算与内存开销过高导致的实用性挑战。为突破这些限制，我们提出HyPHEN——一种深度HCNN架构，其融合了新型卷积算法（RAConv和CAConv）、数据打包方法（2D间隙打包与PRCR方案）以及针对HCNN定制优化的技术。这些改进使HyPHEN显著降低了内存占用及密文旋转、自举等高开销同态操作的次数。最终，HyPHEN将HCNN在CIFAR-10上的推理时延降至实用水平（ResNet-20：1.4秒），并首次在ImageNet上实现HCNN推理（ResNet-18：14.7秒）。