The subject of green AI has been gaining attention within the deep learning community given the recent trend of ever larger and more complex neural network models. Existing solutions for reducing the computational load of training at inference time usually involve pruning the network parameters. Pruning schemes often create extra overhead either by iterative training and fine-tuning for static pruning or repeated computation of a dynamic pruning graph. We propose a new parameter pruning strategy for learning a lighter-weight sub-network that minimizes the energy cost while maintaining comparable performance to the fully parameterised network on given downstream tasks. Our proposed pruning scheme is green-oriented, as it only requires a one-off training to discover the optimal static sub-networks by dynamic pruning methods. The pruning scheme consists of a binary gating module and a novel loss function to uncover sub-networks with user-defined sparsity. Our method enables pruning and training simultaneously, which saves energy in both the training and inference phases and avoids extra computational overhead from gating modules at inference time. Our results on CIFAR-10 and CIFAR-100 suggest that our scheme can remove 50% of connections in deep networks with 1% reduction in classification accuracy. Compared to other related pruning methods, our method demonstrates a lower drop in accuracy for equivalent reductions in computational cost.

翻译：绿色人工智能（Green AI）近年来在深度学习领域备受关注，其核心动因在于神经网络模型趋于庞杂化。现有降低推理阶段计算负荷的方案多采用参数剪枝技术，但静态剪枝需迭代训练与微调产生额外开销，动态剪枝则需重复计算剪枝图。本文提出一种新型参数剪枝策略，旨在通过学习轻量化子网络，在保持与全参数化网络相当性能的前提下最小化下游任务能耗。该方案具有绿色导向特性，仅需一次性训练即可通过动态剪枝方法发现最优静态子网络。剪枝框架由二值化门控模块与新型损失函数构成，可生成具有用户定义稀疏度的子网络。本方法实现剪枝与训练的同步进行，既节省训练与推理阶段的能耗，又规避推理时门控模块带来的额外计算开销。在CIFAR-10与CIFAR-100数据集上的实验表明，本方案可移除深度网络50%的连接，而分类准确率仅下降1%。与同类剪枝方法相比，在同等计算成本缩减幅度下，本方法的准确率降幅更低。