The utilisation of Deep Learning (DL) is advancing into increasingly more sophisticated applications. While it shows great potential to provide transformational capabilities, DL also raises new challenges regarding its reliability in critical functions. In this paper, we present a model-agnostic reliability assessment method for DL classifiers, based on evidence from robustness evaluation and the operational profile (OP) of a given application. We partition the input space into small cells and then "assemble" their robustness (to the ground truth) according to the OP, where estimators on the cells' robustness and OPs are provided. Reliability estimates in terms of the probability of misclassification per input (pmi) can be derived together with confidence levels. A prototype tool is demonstrated with simplified case studies. Model assumptions and extension to real-world applications are also discussed. While our model easily uncovers the inherent difficulties of assessing the DL dependability (e.g. lack of data with ground truth and scalability issues), we provide preliminary/compromised solutions to advance in this research direction.

翻译：深层学习(DL)的利用正在逐步进入日益复杂的应用领域。虽然DL显示出提供转化能力的巨大潜力,但它也在其关键功能的可靠性方面提出了新的挑战。在本文件中,我们根据可靠度评估和特定应用的操作概况(OP)中的证据,为DL分类者提出了一个模型-不可知性可靠性评估方法。我们根据OP,将输入空间分成小细胞,然后根据OP,将输入空间“组合”其(对地面真相)的强度(对地面真相),提供细胞坚固度和OP的估测者。对于每项投入(pmi)分类错误概率的可靠性估计可以与信任度一起得出。一个原型工具通过简化的案例研究得到证明。模型假设和对现实世界应用的推广也被讨论。虽然我们的模型很容易发现评估DL可靠性的内在困难(例如缺乏具有地面真相和可缩放度问题的数据),但我们为推进这一研究方向提供了初步/比较的解决方案。