Deep Neural Networks (DNN) are nowadays largely adopted in many application domains thanks to their human-like, or even superhuman, performance in specific tasks. However, due to unpredictable/unconsidered operating conditions, unexpected failures show up on field, making the performance of a DNN in operation very different from the one estimated prior to release. In the life cycle of DNN systems, the assessment of accuracy is typically addressed in two ways: offline, via sampling of operational inputs, or online, via pseudo-oracles. The former is considered more expensive due to the need for manual labeling of the sampled inputs. The latter is automatic but less accurate. We believe that emerging iterative industrial-strength life cycle models for Machine Learning systems, like MLOps, offer the possibility to leverage inputs observed in operation not only to provide faithful estimates of a DNN accuracy, but also to improve it through remodeling/retraining actions. We propose DAIC (DNN Assessment and Improvement Cycle), an approach which combines ''low-cost'' online pseudo-oracles and ''high-cost'' offline sampling techniques to estimate and improve the operational accuracy of a DNN in the iterations of its life cycle. Preliminary results show the benefits of combining the two approaches and integrating them in the DNN life cycle.

翻译：深度神经网络（DNN）如今在众多应用领域广泛采用，这得益于其在特定任务中表现出类人甚至超越人类的性能。然而，由于不可预测或未考虑的运行条件，现场会出现意外故障，导致DNN在运行中的性能与发布前评估的性能差异显著。在DNN系统的生命周期中，准确度评估通常通过两种方式解决：离线方式（通过采样运行输入）或在线方式（通过伪预言机）。前者因需要对采样输入进行人工标注而更昂贵，后者虽自动化但准确度较低。我们认为，机器学习系统（如MLOps）新兴的工业级迭代生命周期模型，为利用运行中观测到的输入提供了可能，不仅能对DNN准确度进行可靠估计，还能通过重构/再训练操作提升准确度。我们提出DAIC（DNN评估与改进周期）方法，将"低成本"在线伪预言机与"高成本"离线采样技术相结合，在DNN生命周期迭代中估计并改进其运行准确度。初步结果表明，将这两种方法结合并集成到DNN生命周期中具有显著优势。