Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle.

翻译：设计兼具表达能力且保持任务已知不变性的模型日益困难。现有解决方案以计算或内存资源为代价换取不变性。本研究展示如何利用随机性设计既具表达性又具不变性且资源消耗更少的模型。受随机算法启发，我们的核心见解是：接受普适逼近和不变性的概率概念可减少资源需求。具体而言，我们提出一类称为随机线性分类器（RLCs）的二分类模型。我们给出了参数和样本量的条件，在此条件下RLCs能够高概率逼近任意（光滑）函数，同时保持对紧致群变换的不变性。基于此结果，我们设计了三种RLCs，它们可证明地在集合、图和球面数据的分类任务中实现概率不变性。我们展示了这些模型相比（确定性）神经网络及其不变性对应模型，能以更少资源实现概率不变性和普适性。最后，我们在已知确定性不变神经网络难以处理的不变性任务上，通过实验论证了这类新模型的优势。