Adversarial nets have proved to be powerful in various domains including generative modeling (GANs), transfer learning, and fairness. However, successfully training adversarial nets using first-order methods remains a major challenge. Typically, careful choices of the learning rates are needed to maintain the delicate balance between the competing networks. In this paper, we design a novel learning rate scheduler that dynamically adapts the learning rate of the adversary to maintain the right balance. The scheduler is driven by the fact that the loss of an ideal adversarial net is a constant known a priori. The scheduler is thus designed to keep the loss of the optimized adversarial net close to that of an ideal network. We run large-scale experiments to study the effectiveness of the scheduler on two popular applications: GANs for image generation and adversarial nets for domain adaptation. Our experiments indicate that adversarial nets trained with the scheduler are less likely to diverge and require significantly less tuning. For example, on CelebA, a GAN with the scheduler requires only one-tenth of the tuning budget needed without a scheduler. Moreover, the scheduler leads to statistically significant improvements in model quality, reaching up to $27\%$ in Frechet Inception Distance for image generation and $3\%$ in test accuracy for domain adaptation.

翻译：对抗网络已被证明在生成建模（GANs）、迁移学习和公平性等多个领域具有强大能力。然而，使用一阶方法成功训练对抗网络仍是一项重大挑战。通常需要精心选择学习率以维持竞争网络之间的微妙平衡。本文设计了一种新颖的学习率调度器，能够动态调整对手网络的学习率以保持适当平衡。该调度器的驱动力来自一个事实：理想对抗网络的损失是先验已知的常数。因此，调度器旨在使优化后的对抗网络损失接近理想网络的损失。我们通过大规模实验研究了该调度器在两种流行应用中的有效性：用于图像生成的GANs和用于领域适应的对抗网络。实验表明，使用该调度器的对抗网络不易发散，且需要显著更少的调参工作。例如，在CelebA数据集上，配备调度器的GAN仅需无调度器时十分之一的调参预算。此外，该调度器还能带来模型质量的统计学显著提升，在图像生成的Frechet Inception距离上提升高达27%，在领域适应的测试准确率上提升3%。