Existing work on understanding deep learning often employs measures that compress all data-dependent information into a few numbers. In this work, we adopt a perspective based on the role of individual examples. We introduce a measure of the computational difficulty of making a prediction for a given input: the (effective) prediction depth. Our extensive investigation reveals surprising yet simple relationships between the prediction depth of a given input and the model's uncertainty, confidence, accuracy and speed of learning for that data point. We further categorize difficult examples into three interpretable groups, demonstrate how these groups are processed differently inside deep models and showcase how this understanding allows us to improve prediction accuracy. Insights from our study lead to a coherent view of a number of separately reported phenomena in the literature: early layers generalize while later layers memorize; early layers converge faster and networks learn easy data and simple functions first.

翻译：了解深层学习的现有工作往往采用将所有依赖数据的信息压缩成几个数字的措施。在这项工作中,我们采用了基于单个实例作用的视角。我们引入了对特定投入作出预测的计算困难度:(有效)预测深度。我们的广泛调查揭示了特定投入的预测深度与该数据点模型的不确定性、信心、准确性和学习速度之间的令人惊讶但又简单的关系。我们进一步将困难的例子分为三个可解释的组别,展示这些组别是如何在深层模型中被不同处理的,并展示了这些组别如何在深层模型中被不同处理,并展示了这种理解使我们能够改进预测准确性。我们研究的见解使我们对文献中一些单独报告的现象有了一致的看法:早期的层层宽化,后来的层混;早期的层汇合速度更快,网络首先学习容易的数据和简单功能。