Deep neural networks obtained by standard training have been constantly plagued by adversarial examples. Although adversarial training demonstrates its capability to defend against adversarial examples, unfortunately, it leads to an inevitable drop in the natural generalization. To address the issue, we decouple the natural generalization and the robust generalization from joint training and formulate different training strategies for each one. Specifically, instead of minimizing a global loss on the expectation over these two generalization errors, we propose a bi-expert framework called \emph{Generalist} where we simultaneously train base learners with task-aware strategies so that they can specialize in their own fields. The parameters of base learners are collected and combined to form a global learner at intervals during the training process. The global learner is then distributed to the base learners as initialized parameters for continued training. Theoretically, we prove that the risks of Generalist will get lower once the base learners are well trained. Extensive experiments verify the applicability of Generalist to achieve high accuracy on natural examples while maintaining considerable robustness to adversarial ones. Code is available at https://github.com/PKU-ML/Generalist.

翻译：摘要：通过标准训练的深度神经网络不断受到对抗样本的困扰。尽管对抗训练展示了抵御对抗样本的能力，但不幸的是，它会导致自然泛化性能的必然下降。为解决这一问题，我们将自然泛化与鲁棒泛化从联合训练中解耦，并为每种泛化分别制定不同的训练策略。具体而言，我们不通过最小化这两种泛化误差的期望来优化全局损失，而是提出一种名为“通用专家”（Generalist）的双专家框架。在该框架中，我们同时训练具有任务感知策略的基础学习器，使其能够专精于各自领域。在训练过程中，基础学习器的参数会定期收集并组合形成全局学习器，随后该全局学习器作为初始化参数重新分发给基础学习器以继续训练。理论上，我们证明了当基础学习器训练充分时，通用专家的风险将进一步降低。大量实验验证了通用专家在保持对自然样本高精度的同时，对对抗样本仍具有显著鲁棒性的适用性。代码发布于 https://github.com/PKU-ML/Generalist。