Generative adversarial networks (GANs) usually struggle in learning from highly diverse data, whose underlying manifold is complex. In this work, we revisit the mathematical foundations of GANs, and theoretically reveal that the native adversarial loss for GAN training is insufficient to fix the problem of subsets with positive Lebesgue measure of the generated data manifold lying out of the real data manifold. Instead, we find that score matching serves as a promising solution to this issue thanks to its capability of persistently pushing the generated data points towards the real data manifold. We thereby propose to improve the optimization of GANs with score matching regularity (SMaRt). Regarding the empirical evidences, we first design a toy example to show that training GANs by the aid of a ground-truth score function can help reproduce the real data distribution more accurately, and then confirm that our approach can consistently boost the synthesis performance of various state-of-the-art GANs on real-world datasets with pre-trained diffusion models acting as the approximate score function. For instance, when training Aurora on the ImageNet 64x64 dataset, we manage to improve FID from 8.87 to 7.11, on par with the performance of one-step consistency model. The source code will be made public.

翻译：生成对抗网络（GANs）在从高度多样化的数据（其底层流形复杂）中学习时通常面临困难。本文中，我们重新审视了GANs的数学基础，并从理论上揭示了原生对抗性损失不足以解决生成数据流形中具有正Lebesgue测度的子集偏离真实数据流形的问题。相反，我们发现分数匹配由于能够持续将生成数据点推向真实数据流形，因此成为该问题的一种有前途的解决方案。为此，我们提出利用分数匹配正则化（SMaRt）来改进GANs的优化。在实证方面，我们首先设计了一个玩具示例，证明借助真实分数函数训练GANs能够更准确地复现真实数据分布；随后确认我们的方法能够通过将预训练扩散模型作为近似分数函数，持续提升多种最先进GANs在真实世界数据集上的合成性能。例如，在ImageNet 64x64数据集上训练Aurora时，我们将FID从8.87提升至7.11，达到了单步一致性模型的性能水平。源代码将公开发布。