In this work, we explore the influence of entropy change in deep learning systems by adding noise to the inputs/latent features. The applications in this paper focus on deep learning tasks within computer vision, but the proposed theory can be further applied to other fields. Noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different learning tasks like image classification and transfer learning. However, this paper aims to rethink whether the conventional proposition always holds. We demonstrate that specific noise can boost the performance of various deep architectures under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the complexity of the task. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieved an unprecedented top 1 accuracy of over 95% on ImageNet. Both theoretical analysis and empirical evidence have confirmed that the presence of positive noise can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us that we can influence the performance of learning systems via information entropy change.

翻译：本文通过向输入/潜在特征添加噪声，探究了深度学习系统中熵变化的影响。本文的应用主要集中于计算机视觉领域的深度学习任务，但所提出的理论可进一步推广至其他领域。噪声通常被认为是对各类深度学习架构（如卷积神经网络和视觉Transformer）及不同学习任务（如图像分类和迁移学习）有害的扰动。然而，本文旨在重新审视这一传统观点是否始终成立。我们证明，特定噪声在特定条件下能够提升各类深度架构的性能。通过降低由信息熵定义的任务复杂度，我们从理论上证明了正噪声带来的性能提升，并在ImageNet等大规模图像数据集上通过实验展示了显著的性能增益。本文使用信息熵来定义任务复杂度，并根据噪声是否有助于降低任务复杂度，将其分为正噪声和有害噪声两类。通过在卷积神经网络和视觉Transformer上开展的大量实验，主动注入正噪声显著提升了模型性能，其中在ImageNet上实现了前所未有的超过95%的Top-1准确率。理论分析与实验结果均证实，正噪声的存在有利于学习过程，而传统认知中的有害噪声确实会损害深度学习模型。噪声的不同角色为特定任务下的深度模型提供了新的解释，并为提升模型性能提供了新范式。此外，这一结果提醒我们，可以通过改变信息熵来影响学习系统的性能。