The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations. In particular, it suggests that there is no benefit in enhancing the signal or encoding it before addressing a classification problem. This assertion can be proven to be true for the case of the optimal Bayes classifier. However, in practice, it is common to perform "low-level" tasks before "high-level" downstream tasks despite the overwhelming capabilities of modern deep neural networks. In this paper, we aim to understand when and why low-level processing can be beneficial for classification. We present a comprehensive theoretical study of a binary classification setup, where we consider a classifier that is tightly connected to the optimal Bayes classifier and converges to it as the number of training samples increases. We prove that for any finite number of training samples, there exists a pre-classification processing that improves the classification accuracy. We also explore the effect of class separation, training set size, and class balance on the relative gain from this procedure. We support our theory with an empirical investigation of the theoretical setup. Finally, we conduct an empirical study where we investigate the effect of denoising and encoding on the performance of practical deep classifiers on benchmark datasets. Specifically, we vary the size and class distribution of the training set, and the noise level, and demonstrate trends that are consistent with our theoretical results.

翻译：数据加工不等式是一条信息论原理，指出信号的信息含量无法通过处理观测结果而增加。具体而言，该原理表明，在处理分类问题之前，增强信号或对其进行编码并无益处。这一论断对最优贝叶斯分类器的情况可证明成立。然而在实际中，尽管现代深度神经网络能力超群，仍在"高层"下游任务之前普遍执行"低层"任务。本文旨在理解低层处理何时及为何能对分类任务产生裨益。我们针对二分类设置展开全面理论研究，其中考虑一种与最优贝叶斯分类器紧密关联且随训练样本数增加收敛于该分类器的分类器。我们证明：对于任意有限训练样本数，总存在一种预分类处理可提升分类准确率。我们还探讨了类别分离度、训练集规模和类别平衡性对该过程相对收益的影响。我们通过理论设置下的实证研究支持理论分析，并最终开展实证研究，探究去噪和编码对实用深度分类器在基准数据集上性能的影响。具体而言，我们改变了训练集的规模、类别分布及噪声水平，并展示了与理论结果一致的实验趋势。