Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle.

翻译：设计既具有表达力又保持任务已知不变性的模型正面临日益严峻的挑战。现有解决方案在不变性与计算或内存资源之间进行权衡。本文展示如何利用随机性设计既具表达力又保持不变性且资源消耗更少的模型。受随机算法启发，我们的核心洞见在于：接受普适近似与不变性的概率化概念可降低资源需求。具体而言，我们提出一类称为随机线性分类器（RLCs）的二分类模型。我们给出了参数与样本量条件，在此条件下RLCs能以高概率逼近任意（光滑）函数，同时保持对紧致群变换的不变性。基于这一结果，我们设计了三种RLCs，分别针对集合、图与球面数据的分类任务，可证明实现概率不变性。我们展示了这些模型如何以比（确定性）神经网络及其不变性对应模型更少的资源实现概率不变性与普适性。最后，我们在确定性不变神经网络已知难以处理的不变性任务上，实证展示了这类新模型的优势。