The well-designed structures in neural networks reflect the prior knowledge incorporated into the models. However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD. In this paper, we propose to incorporate model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters. Such a methodology is referred to as Gradient Re-parameterization, and the optimizers are named RepOptimizers. For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models. From a practical perspective, RepOpt-VGG is a favorable base model because of its simple structure, high inference speed and training efficiency. Compared to Structural Re-parameterization, which adds priors into models via constructing extra training-time structures, RepOptimizers require no extra forward/backward computations and solve the problem of quantization. We hope to spark further research beyond the realms of model structure design. Code and models \url{https://github.com/DingXiaoH/RepOptimizers}.

翻译：神经网络中精心设计的结构反映了模型所融入的先验知识。然而，尽管不同模型具有各异的先验，我们仍习惯于使用与模型无关的优化器（如SGD）来训练它们。本文提出通过根据一组模型特定的超参数修改梯度，将模型特定的先验知识融入优化器中。这种方法被称为梯度重新参数化，相应的优化器命名为RepOptimizers。为追求模型结构的极致简洁，我们聚焦于VGG风格的朴素模型，并展示了使用RepOptimizer训练的此类简单模型（称为RepOpt-VGG）性能与近期精心设计的模型相当或更优。从实际应用角度出发，RepOpt-VGG因其结构简单、推理速度快及训练效率高，是一种理想的基础模型。相较于通过构建额外训练时结构将先验引入模型的结构重新参数化方法，RepOptimizers无需额外的前向/反向计算，并解决了量化问题。我们期望这能激发超越模型结构设计领域的研究。代码和模型见 \url{https://github.com/DingXiaoH/RepOptimizers}。