Due to the unprecedented success of deep learning, it has become an integral component in several multimedia computing applications in todays world. Unfortunately, deep learning systems are not perfect and can fail, sometimes abruptly, without prior warning or explanation. While reducing the error rate of deep neural networks has been the primary focus of the multimedia community, the problem of predicting when a deep learning system is going to fail has received significantly less research attention. In this paper, we propose a simple yet effective framework, MetaErr, to address this under-explored problem in deep learning research. We train a meta-model whose goal is to predict whether a base deep neural network will succeed or fail in predicting a particular data sample, by observing the base models performance on a given learning task. The meta-model is completely agnostic of the architecture and training parameters of the base model. Such an error prediction system can be immensely useful in a variety of smart multimedia applications. Our empirical studies corroborate the promise and potential of our framework against competing baselines. We further demonstrate the usefulness of our framework to improve the performance of pseudo-labeling-based semi-supervised learning, and show that MetaErr outperforms several strong baselines on three benchmark computer vision datasets.

翻译：摘要：鉴于深度学习取得的空前成功，它已成为当今世界多种多媒体计算应用中不可或缺的组成部分。然而，深度学习系统并非完美无缺，有时会在无预警或解释的情况下突然失效。尽管降低深度神经网络的错误率一直是多媒体社区关注的核心问题，但对于预测深度学习系统何时会失败的研究却相对匮乏。本文提出了一个名为MetaErr的简洁而有效的框架，旨在解决深度学习中这一尚未充分探索的问题。我们训练了一个元模型，其目标是通过观察基模型在特定学习任务上的表现，来预测基深度神经网络在预测特定数据样本时是否会成功或失败。该元模型完全独立于基模型的架构及训练参数。这类错误预测系统在多种智能多媒体应用中具有极大的实用性。我们的实验验证了该框架相对于竞争基线的潜力与前景。此外，我们进一步展示了该框架在提升基于伪标签的半监督学习性能方面的有效性，并证明MetaErr在三个基准计算机视觉数据集上优于多个强基线方法。