Neural network certification methods heavily rely on convex relaxations to provide robustness guarantees. However, these relaxations are often imprecise: even the most accurate single-neuron relaxation is incomplete for general ReLU networks, a limitation known as the *single-neuron convex barrier*. While multi-neuron relaxations have been heuristically applied to address this issue, two central questions arise: (i) whether they overcome the convex barrier, and if not, (ii) whether they offer theoretical capabilities beyond those of single-neuron relaxations. In this work, we present the first rigorous analysis of the expressiveness of multi-neuron relaxations. Perhaps surprisingly, we show that they are inherently incomplete, even when allocated sufficient resources to capture finitely many neurons and layers optimally. This result extends the single-neuron barrier to a *universal convex barrier* for neural network certification. On the positive side, we show that completeness can be achieved by either (i) augmenting the network with a polynomial number of carefully designed ReLU neurons or (ii) partitioning the input domain into convex sub-polytopes, thereby distinguishing multi-neuron relaxations from single-neuron ones which are unable to realize the former and have worse partition complexity for the latter. Our findings establish a foundation for multi-neuron relaxations and point to new directions for certified robustness, including training methods tailored to multi-neuron relaxations and verification methods with multi-neuron relaxations as the main subroutine.

翻译：神经网络认证方法严重依赖凸松弛来提供鲁棒性保证。然而，这些松弛通常并不精确：即使是最精确的单神经元松弛对于一般的ReLU网络也是不完备的，这一局限性被称为*单神经元凸障碍*。虽然多神经元松弛已被启发式地应用于解决此问题，但两个核心问题随之产生：(i) 它们是否克服了凸障碍，以及如果未能克服，(ii) 它们是否提供了超越单神经元松弛的理论能力。在本工作中，我们首次对多神经元松弛的表达能力进行了严格分析。或许令人惊讶的是，我们证明了它们本质上是完备的，即使分配了足够的资源来最优地捕获有限多个神经元和层。这一结果将单神经元障碍扩展为神经网络认证的*普适凸障碍*。从积极的一面来看，我们证明了完备性可以通过以下任一方式实现：(i) 为网络增广多项式数量级、精心设计的ReLU神经元，或(ii) 将输入域划分为凸子多面体，从而将多神经元松弛与单神经元松弛区分开来——后者无法实现前者，且对于后者的划分复杂度更差。我们的发现为多神经元松弛奠定了基础，并为认证鲁棒性指明了新方向，包括针对多神经元松弛定制的训练方法，以及以多神经元松弛为主要子程序的验证方法。