We study minimax optimization problems defined over infinite-dimensional function classes. In particular, we restrict the functions to the class of overparameterized two-layer neural networks and study (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural network. As an initial step, we consider the minimax optimization problem stemming from estimating a functional equation defined by conditional expectations via adversarial estimation, where the objective function is quadratic in the functional space. For this problem, we establish convergence under the mean-field regime by considering the continuous-time and infinite-width limit of the optimization dynamics. Under this regime, gradient descent-ascent corresponds to a Wasserstein gradient flow over the space of probability measures defined over the space of neural network parameters. We prove that the Wasserstein gradient flow converges globally to a stationary point of the minimax objective at a $\mathcal{O}(T^{-1} + \alpha^{-1} ) $ sublinear rate, and additionally finds the solution to the functional equation when the regularizer of the minimax objective is strongly convex. Here $T$ denotes the time and $\alpha$ is a scaling parameter of the neural network. In terms of representation learning, our results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $\mathcal{O}(\alpha^{-1})$, measured in terms of the Wasserstein distance. Finally, we apply our general results to concrete examples including policy evaluation, nonparametric instrumental variable regression, and asset pricing.

翻译：我们研究定义在无穷维函数类上的极小极大优化问题。具体而言，我们将函数限制在过参数化两层神经网络的函数类中，研究（i）梯度下降-上升算法的收敛性以及（ii）神经网络的表示学习。作为初始步骤，我们考虑通过对抗估计估计由条件期望定义的函数方程所衍生的极小极大优化问题，其中目标函数在函数空间中是二次的。针对该问题，我们通过考虑优化动力学的连续时间与无穷宽度极限，建立了平均场体制下的收敛性。在该体制下，梯度下降-上升对应于定义在神经网络参数空间上的概率测度空间中的Wasserstein梯度流。我们证明Wasserstein梯度流以$\mathcal{O}(T^{-1} + \alpha^{-1} )$次线性速率全局收敛到极小极大目标的一个驻点，并且当极小极大目标的正则化项强凸时，还能额外求得函数方程的解。这里$T$表示时间，$\alpha$是神经网络的缩放参数。在表示学习方面，我们的结果表明，神经网络诱导的特征表示允许以Wasserstein距离度量的$\mathcal{O}(\alpha^{-1})$量级偏离初始表示。最后，我们将通用结果应用于具体实例，包括策略评估、非参数工具变量回归以及资产定价。