Training deep neural networks (DNNs) in low-dimensional subspaces is a promising direction for achieving efficient training and better generalization performance. Our previous work extracts the subspaces by performing the dimension reduction method over the training trajectory, which verifies that DNN could be well-trained in a tiny subspace. However, that method is inefficient for subspace extraction and numerically unstable, limiting its applicability to more general tasks. In this paper, we connect subspace training to weight averaging and propose \emph{Trainable Weight Averaging} (TWA), a general approach for subspace training. TWA is efficient in terms of subspace extraction and easy to use, making it a promising new optimizer for DNN's training. Our design also includes an efficient scheme that allows parallel training across multiple nodes to handle large-scale problems and evenly distribute the memory and computation burden to each node. TWA can be used for both efficient training and generalization enhancement, for different neural network architectures, and for various tasks from image classification and object detection, to neural language processing. The code of implementation is available at https://github.com/nblt/TWA, which includes extensive experiments covering various benchmark computer vision and neural language processing tasks with various architectures.

翻译：在低维子空间中训练深度神经网络（DNN）是实现高效训练与更好泛化性能的有前景方向。我们先前的工作通过沿训练轨迹执行降维方法提取子空间，验证了DNN可在极小子空间中得到良好训练。然而，该方法在子空间提取上效率低下且数值不稳定，限制了其在更通用任务中的适用性。本文我们将子空间训练与权重平均相联系，提出可训练权重平均（TWA）——一种通用的子空间训练方法。TWA在子空间提取方面高效且易于使用，使其成为极具潜力的DNN训练优化器。我们的设计还包含一种高效方案，允许跨多节点并行训练以处理大规模问题，并将内存与计算负担均匀分布至各节点。TWA既可用于高效训练，也可用于泛化增强，适用于不同神经网络架构，以及从图像分类、目标检测到神经语言处理等多种任务。实现代码发布于https://github.com/nblt/TWA，其中包含覆盖多种架构的基准计算机视觉与神经语言处理任务的广泛实验。