Deep learning neural network models must be large enough to adapt to their problem domain, while small enough to avoid overfitting training data during gradient descent. To balance these competing demands, overprovisioned deep learning models such as transformers are trained for a single epoch on large data sets, and hence inefficient with both computing resources and training data. In response to these inefficiencies, we exploit learning theory to derive Occam Gradient Descent, an algorithm that interleaves adaptive reduction of model size to minimize generalization error, with gradient descent on model weights to minimize fitting error. In contrast, traditional gradient descent greedily minimizes fitting error without regard to generalization error. Our algorithm simultaneously descends the space of weights and topological size of any neural network without modification, and is effective in our experiments in outperforming traditional gradient descent with or without post-train pruning in accuracy, compute and model compression.

翻译：深度学习神经网络模型必须足够大以适应其问题领域，同时足够小以避免在梯度下降过程中对训练数据过拟合。为平衡这些相互矛盾的需求，过度配置的深度学习模型（如Transformer）通常在大规模数据集上仅训练单个周期，从而在计算资源和训练数据方面均存在效率低下的问题。针对这些低效现象，我们运用学习理论推导出奥卡姆梯度下降法——该算法通过交替执行模型规模的自适应缩减（以最小化泛化误差）和模型权重的梯度下降（以最小化拟合误差）来实现优化。相比之下，传统梯度下降法仅贪婪地最小化拟合误差而忽略泛化误差。我们的算法无需修改即可同时优化任意神经网络的权重空间与拓扑规模，实验证明其在精度、计算效率和模型压缩方面均优于传统梯度下降法（无论是否进行训练后剪枝）。