Recently, there has been tremendous interest in industry 4.0 infrastructure to address labor shortages in global supply chains. Deploying artificial intelligence-enabled robotic bin picking systems in real world has become particularly important for reducing stress and physical demands of workers while increasing speed and efficiency of warehouses. To this end, artificial intelligence-enabled robotic bin picking systems may be used to automate order picking, but with the risk of causing expensive damage during an abnormal event such as sensor failure. As such, reliability becomes a critical factor for translating artificial intelligence research to real world applications and products. In this paper, we propose a reliable object detection and segmentation system with MultiModal Redundancy (MMRNet) for tackling object detection and segmentation for robotic bin picking using data from different modalities. This is the first system that introduces the concept of multimodal redundancy to address sensor failure issues during deployment. In particular, we realize the multimodal redundancy framework with a gate fusion module and dynamic ensemble learning. Finally, we present a new label-free multi-modal consistency (MC) score that utilizes the output from all modalities to measure the overall system output reliability and uncertainty. Through experiments, we demonstrate that in an event of missing modality, our system provides a much more reliable performance compared to baseline models. We also demonstrate that our MC score is a more reliability indicator for outputs during inference time compared to the model generated confidence scores that are often over-confident.

翻译：近年来，工业4.0基础设施在应对全球供应链劳动力短缺方面引起了极大关注。在现实世界中部署基于人工智能的机器人料箱抓取系统，对于减轻工人的压力和体力负担、同时提升仓库的速度与效率尤为重要。为此，基于人工智能的机器人料箱抓取系统可用于实现自动订单拣选，但在传感器故障等异常事件发生时存在造成昂贵损坏的风险。因此，可靠性成为将人工智能研究转化为现实应用与产品的关键因素。本文提出了一种具备多模态冗余的可靠目标检测与分割系统（MMRNet），用于利用不同模态数据解决机器人料箱抓取中的目标检测与分割问题。这是首个引入多模态冗余概念以应对部署期间传感器故障问题的系统。具体而言，我们通过门控融合模块与动态集成学习实现了多模态冗余框架。最后，我们提出了一种新的无标签多模态一致性（MC）分数，该分数利用所有模态的输出来衡量系统整体输出的可靠性与不确定性。实验表明，在模态缺失的情况下，我们的系统相比基线模型提供了更可靠的性能。我们还证明，与模型生成的往往过度自信的置信度分数相比，我们的MC分数是推理期间输出更可靠的指标。