Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We show examples of potential applications to the analysis of deep-learning systems.

翻译：人类习惯于包含规律性和例外的环境。例如,在大多数加油站,在抽水前支付一次,但偶尔的农村站不接受事先付款。同样,深神经网络可以对共同模式或结构的事例进行概括,但有能力对稀有或不正常的形式进行记忆。我们通过一致性评分分析单个事例如何用模型处理。评分是按数据分布不同规模抽样的培训组合所显示的留置实例的预期准确性。我们从多个数据集中获取了个别案例的这一评分的经验估计,我们显示得分在连续体的一端发现分配之外和贴错标签的例子,而在另一端则非常常见的例子。我们利用培训期间收集的统计数据,用计算成本低的方法确定一致性评分的比重。我们展示了深学习系统分析的潜在应用实例。