We study the interpolation, or memorization, power of deep ReLU neural networks. Specifically, we consider the question of how efficiently, in terms of the number of parameters, deep ReLU networks can interpolate values at $N$ datapoints in the unit ball which are separated by a distance $\delta$. We show that $\Omega(N)$ parameters are required in the regime where $\delta$ is exponentially small in $N$, which gives the sharp result in this regime since $O(N)$ parameters are always sufficient. This also shows that the bit-extraction technique used to prove lower bounds on the VC dimension cannot be applied to irregularly spaced datapoints.

翻译：我们研究了深度ReLU神经网络的插值（或记忆）能力。具体而言，我们探讨了深度ReLU网络在效率方面（以参数数量衡量）如何能够插值单位球内间距为$\delta$的$N$个数据点的数值。我们证明，在$\delta$相对于$N$呈指数级小的情境下，需要$\Omega(N)$个参数，这给出了该情境下的尖锐结果，因为$O(N)$个参数总是足够的。这也表明，用于证明VC维下界的位提取技术无法应用于不规则间隔的数据点。