As a booming research area in the past decade, deep learning technologies have been driven by big data collected and processed on an unprecedented scale. However, privacy concerns arise due to the potential leakage of sensitive information from the training data. Recent research has revealed that deep learning models are vulnerable to various privacy attacks, including membership inference attacks, attribute inference attacks, and gradient inversion attacks. Notably, the efficacy of these attacks varies from model to model. In this paper, we answer a fundamental question: Does model architecture affect model privacy? By investigating representative model architectures from CNNs to Transformers, we demonstrate that Transformers generally exhibit higher vulnerability to privacy attacks compared to CNNs. Additionally, We identify the micro design of activation layers, stem layers, and LN layers, as major factors contributing to the resilience of CNNs against privacy attacks, while the presence of attention modules is another main factor that exacerbates the privacy vulnerability of Transformers. Our discovery reveals valuable insights for deep learning models to defend against privacy attacks and inspires the research community to develop privacy-friendly model architectures.

翻译：作为过去十年中蓬勃发展的研究领域，深度学习技术依赖于以空前规模收集和处理的大数据。然而，训练数据中潜在敏感信息的泄露引发了隐私担忧。最新研究表明，深度学习模型易受多种隐私攻击，包括成员推断攻击、属性推断攻击和梯度反转攻击。值得注意的是，这些攻击的有效性因模型而异。本文回答了一个核心问题：模型架构是否会影响模型隐私？通过研究从CNN到Transformer的代表性模型架构，我们证明Transformer通常比CNN更容易受到隐私攻击。此外，我们识别出激活层、茎层和层归一化（LN）层的微观设计是CNN对抗隐私攻击的主要韧性因素，而注意力模块的存在则是加剧Transformer隐私脆弱性的另一主要因素。我们的发现为深度学习模型防御隐私攻击提供了宝贵见解，并激励研究社区开发隐私友好的模型架构。