Classes of target functions containing a large number of approximately orthogonal elements are known to be hard to learn by the Statistical Query algorithms. Recently this classical fact re-emerged in a theory of gradient-based optimization of neural networks. In the novel framework, the hardness of a class is usually quantified by the variance of the gradient with respect to a random choice of a target function. A set of functions of the form $x\to ax \bmod p$, where $a$ is taken from ${\mathbb Z}_p$, has attracted some attention from deep learning theorists and cryptographers recently. This class can be understood as a subset of $p$-periodic functions on ${\mathbb Z}$ and is tightly connected with a class of high-frequency periodic functions on the real line. We present a mathematical analysis of limitations and challenges associated with using gradient-based learning techniques to train a high-frequency periodic function or modular multiplication from examples. We highlight that the variance of the gradient is negligibly small in both cases when either a frequency or the prime base $p$ is large. This in turn prevents such a learning algorithm from being successful.

翻译：已知包含大量近似正交元素的目标函数类难以通过统计查询算法学习。近期，这一经典结论重新出现在基于梯度的神经网络优化理论中。在这一新型框架下，目标函数类的难度通常通过随机选择目标函数时梯度的方差来量化。形如$x\to ax \bmod p$（其中$a$取自${\mathbb Z}_p$）的函数集近期引起了深度学习理论家与密码学家的关注。该类函数可理解为整数集${\mathbb Z}$上$p$周期函数的子集，并与实数轴上高频周期函数类密切相关。本文对基于梯度的学习技术在训练高频周期函数或模乘运算时存在的局限与挑战进行了数学分析。我们指出，当频率或素数底数$p$较大时，这两种情况下的梯度方差均极小，从而阻碍此类学习算法的成功。