This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparameterized two-layer neural networks. In particular, we consider the minimax optimization problem stemming from estimating linear functional equations defined by conditional expectations, where the objective functions are quadratic in the functional spaces. We address (i) the convergence of the stochastic gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. We establish convergence under the mean-field regime by considering the continuous-time and infinite-width limit of the optimization dynamics. Under this regime, the stochastic gradient descent-ascent corresponds to a Wasserstein gradient flow over the space of probability measures defined over the space of neural network parameters. We prove that the Wasserstein gradient flow converges globally to a stationary point of the minimax objective at a $O(T^{-1} + \alpha^{-1})$ sublinear rate, and additionally finds the solution to the functional equation when the regularizer of the minimax objective is strongly convex. Here $T$ denotes the time and $\alpha$ is a scaling parameter of the neural networks. In terms of representation learning, our results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(\alpha^{-1})$, measured in terms of the Wasserstein distance. Finally, we apply our general results to concrete examples including policy evaluation, nonparametric instrumental variable regression, asset pricing, and adversarial Riesz representer estimation.

翻译：本文研究定义于过参数化双层神经网络无限维函数类上的极小极大优化问题。具体而言，我们考虑源于条件期望定义的线性函数方程估计所产生的极小极大优化问题，其目标函数在函数空间中是二次型的。我们重点探讨（i）随机梯度下降-上升算法的收敛性，以及（ii）神经网络表示学习特性。通过考虑优化动态的连续时间与无限宽度极限，我们在平均场体系下建立了收敛性理论。在此体系下，随机梯度下降-上升算法对应于定义在神经网络参数空间上的概率测度空间中的Wasserstein梯度流。我们证明该Wasserstein梯度流以$O(T^{-1} + \alpha^{-1})$次线性速率全局收敛至极小极大目标的驻点，且当极小极大目标的正则项强凸时能收敛至函数方程的解。其中$T$表示时间，$\alpha$为神经网络的尺度参数。在表示学习方面，我们的结果表明神经网络诱导的特征表示允许以$O(\alpha^{-1})$量级偏离初始表示（以Wasserstein距离度量）。最后，我们将理论结果应用于策略评估、非参数工具变量回归、资产定价及对抗性Riesz表示子估计等具体案例。