Extraction of structure, in particular of group symmetries, is increasingly crucial to understanding and building intelligent models. In particular, some information-theoretic models of parsimonious learning have been argued to induce invariance extraction. Here, we formalise these arguments from a group-theoretic perspective. We then extend them to the study of more general probabilistic symmetries, through compressions preserving well-studied geometric measures of complexity. More precisely, we formalise a trade-off between compression and preservation of the divergence from a given hierarchical model, yielding a novel generalisation of the Information Bottleneck framework. Through appropriate choices of hierarchical models, we fully characterise (in the discrete and full support case) channel invariance, channel equivariance and distribution invariance under permutation. Allowing imperfect divergence preservation then leads to principled definitions of "soft symmetries", where the "coarseness" corresponds to the degree of compression of the system. In simple synthetic experiments, we demonstrate that our method successively recovers, at increasingly compressed "resolutions", nested but increasingly perturbed equivariances, where new equivariances emerge at bifurcation points of the trade-off parameter. Our framework suggests a new path for the extraction of generalised probabilistic symmetries.

翻译：结构提取，特别是群对称性的提取，对于理解和构建智能模型日益关键。具体而言，一些基于信息论的简约学习模型被认为能够诱导不变性提取。本文从群论视角对这些论点进行了形式化。随后，我们通过保留经过深入研究的几何复杂度度量，将这些论点推广到更一般的概率对称性研究中。更精确地说，我们形式化了压缩与保持给定层次模型散度之间的权衡，从而提出了一种信息瓶颈框架的新颖推广。通过适当选择层次模型，我们完整刻画了（在离散且全支撑情况下）置换下的通道不变性、通道等变性与分布不变性。允许不完美的散度保持则引出了"软对称性"的原则性定义，其中"粗糙度"对应于系统压缩的程度。在简单的合成实验中，我们证明了我们的方法能够在逐渐压缩的"分辨率"下，依次恢复嵌套但扰动逐渐增大的等变性，新的等变性在权衡参数的分岔点处涌现。我们的框架为提取广义概率对称性提出了一条新路径。