In energy-efficient schemes, finding the optimal size of deep learning models is very important and has a broad impact. Meanwhile, recent studies have reported an unexpected phenomenon, the sparse double descent: as the model's sparsity increases, the performance first worsens, then improves, and finally deteriorates. Such a non-monotonic behavior raises serious questions about the optimal model's size to maintain high performance: the model needs to be sufficiently over-parametrized, but having too many parameters wastes training resources. In this paper, we aim to find the best trade-off efficiently. More precisely, we tackle the occurrence of the sparse double descent and present some solutions to avoid it. Firstly, we show that a simple $\ell_2$ regularization method can help to mitigate this phenomenon but sacrifices the performance/sparsity compromise. To overcome this problem, we then introduce a learning scheme in which distilling knowledge regularizes the student model. Supported by experimental results achieved using typical image classification setups, we show that this approach leads to the avoidance of such a phenomenon.

翻译：在节能方案中，寻找深度学习模型的最优规模至关重要且具有广泛影响。与此同时，近期研究报道了一个令人意外的现象——稀疏双下降：随着模型稀疏度增加，性能先恶化、后改善、最终再度衰退。这种非单调行为引发了关于维持高性能所需最优模型规模的深刻疑问：模型需要充分过参数化，但参数过多又会浪费训练资源。本文旨在高效寻求最佳权衡方案。更具体地说，我们针对稀疏双下降现象进行剖析，并提出避免该现象的若干解决方案。首先，我们证明简单的$\ell_2$正则化方法虽能缓解该现象，却会损害性能与稀疏度的权衡。为克服这一局限，我们进一步引入一种学习范式，通过知识蒸馏对学生模型进行正则化。基于典型图像分类任务的实验结果，我们证实该方法能够有效避免该现象的发生。