In recent years, Deep Learning models have shown a great performance in complex optimization problems. They generally require large training datasets, which is a limitation in most practical cases. Transfer learning allows importing the first layers of a pre-trained architecture and connecting them to fully-connected layers to adapt them to a new problem. Consequently, the configuration of the these layers becomes crucial for the performance of the model. Unfortunately, the optimization of these models is usually a computationally demanding task. One strategy to optimize Deep Learning models is the pruning scheme. Pruning methods are focused on reducing the complexity of the network, assuming an expected performance penalty of the model once pruned. However, the pruning could potentially be used to improve the performance, using an optimization algorithm to identify and eventually remove unnecessary connections among neurons. This work proposes EvoPruneDeepTL, an evolutionary pruning model for Transfer Learning based Deep Neural Networks which replaces the last fully-connected layers with sparse layers optimized by a genetic algorithm. Depending on its solution encoding strategy, our proposed model can either perform optimized pruning or feature selection over the densely connected part of the neural network. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show the contribution of EvoPruneDeepTL and feature selection to the overall computational efficiency of the network as a result of the optimization process. In particular, the accuracy is improved, reducing at the same time the number of active neurons in the final layers.

翻译：近年来，深度学习模型在复杂优化问题中展现出卓越的性能。这类模型通常需要大规模训练数据集，这在大多数实际应用中构成限制。迁移学习允许导入预训练架构的前几层，并将其与全连接层连接，从而适应新的问题。因此，这些层的配置对模型性能至关重要。然而，这类模型的优化通常是一项计算密集性任务。优化深度学习模型的一种策略是剪枝方案。剪枝方法旨在降低网络复杂度，但通常假设剪枝后模型会承受性能损失。然而，剪枝经过优化算法识别并最终移除神经元间不必要的连接后，反而有可能提升模型性能。本文提出EvoPruneDeepTL——一种用于基于迁移学习的深度神经网络的进化剪枝模型，该模型将最后的全连接层替换为由遗传算法优化的稀疏层。根据其解编码策略，所提模型可针对神经网络的密集连接部分执行优化剪枝或特征选择。我们通过多个数据集开展不同实验以评估该方法的优势。结果表明，EvoPruneDeepTL与特征选择在优化过程中对网络整体计算效率有所贡献。具体而言，模型在提升精度的同时，减少了最终层的活跃神经元数量。