The past decade has witnessed a drastic increase in modern deep neural networks (DNNs) size, especially for generative adversarial networks (GANs). Since GANs usually suffer from high computational complexity, researchers have shown an increased interest in applying pruning methods to reduce the training and inference costs of GANs. Among different pruning methods invented for supervised learning, dynamic sparse training (DST) has gained increasing attention recently as it enjoys excellent training efficiency with comparable performance to post-hoc pruning. Hence, applying DST on GANs, where we train a sparse GAN with a fixed parameter count throughout training, seems to be a good candidate for reducing GAN training costs. However, a few challenges, including the degrading training instability, emerge due to the adversarial nature of GANs. Hence, we introduce a quantity called balance ratio (BR) to quantify the balance of the generator and the discriminator. We conduct a series of experiments to show the importance of BR in understanding sparse GAN training. Building upon single dynamic sparse training (SDST), where only the generator is adjusted during training, we propose double dynamic sparse training (DDST) to control the BR during GAN training. Empirically, DDST automatically determines the density of the discriminator and greatly boosts the performance of sparse GANs on multiple datasets.

翻译：过去十年间，现代深度神经网络（DNN）的规模急剧增长，尤其是生成对抗网络（GANs）。由于GANs通常面临高计算复杂度的挑战，研究人员对应用剪枝方法以减少GANs训练与推理成本的兴趣日益增加。在监督学习领域发明的各类剪枝方法中，动态稀疏训练（DST）近年来备受关注，因其在保持与事后剪枝相当性能的同时，展现出卓越的训练效率。因此，将DST应用于GANs（即在训练过程中以固定参数数量训练稀疏GAN）似乎是降低GAN训练成本的可行方案。然而，GANs的对抗特性引发了一系列挑战，包括训练稳定性下降等问题。为此，我们引入一个名为平衡比（BR）的量化指标，用于衡量生成器与判别器之间的平衡状态。通过一系列实验，我们揭示了BR在理解稀疏GAN训练中的关键作用。在仅调整生成器的单动态稀疏训练（SDST）基础上，我们提出双重动态稀疏训练（DDST），以在GAN训练过程中控制BR。实验结果表明，DDST能自动确定判别器的密度，并显著提升稀疏GAN在多个数据集上的性能。