An autonomous service robot should be able to interact with its environment safely and robustly without requiring human assistance. Unstructured environments are challenging for robots since the exact prediction of outcomes is not always possible. Even when the robot behaviors are well-designed, the unpredictable nature of physical robot-object interaction may prevent success in object manipulation. Therefore, execution of a manipulation action may result in an undesirable outcome involving accidents or damages to the objects or environment. Situation awareness becomes important in such cases to enable the robot to (i) maintain the integrity of both itself and the environment, (ii) recover from failed tasks in the short term, and (iii) learn to avoid failures in the long term. For this purpose, robot executions should be continuously monitored, and failures should be detected and classified appropriately. In this work, we focus on detecting and classifying both manipulation and post-manipulation phase failures using the same exteroception setup. We cover a diverse set of failure types for primary tabletop manipulation actions. In order to detect these failures, we propose FINO-Net [1], a deep multimodal sensor fusion based classifier network. Proposed network accurately detects and classifies failures from raw sensory data without any prior knowledge. In this work, we use our extended FAILURE dataset [1] with 99 new multimodal manipulation recordings and annotate them with their corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80 failure classification F1 scores. Experimental results show that proposed architecture is also appropriate for real-time use.

翻译：自主服务机器人应能在无需人类协助的情况下安全、鲁棒地与环境交互。非结构化环境对机器人构成挑战，因为精确预测操作结果并非总是可行。即使机器人行为设计完善，物理人机交互的不可预测性仍可能阻碍物体操作成功。因此，操作动作的执行可能导致不良后果，包括对物体或环境造成意外损坏。在此类情况下，态势感知至关重要，使机器人能够：(i) 维护自身与环境的完整性，(ii) 在短期内从失败任务中恢复，(iii) 在长期内学习避免失败。为此，需持续监控机器人执行过程，并适当地检测与分类故障。本研究聚焦于使用相同的感知外传感器配置，同时检测与分类操作阶段及操作后阶段的故障。我们涵盖了桌面基础操作任务中多样化的故障类型。为检测这些故障，提出FINO-Net [1]——一种基于深度多模态传感器融合的分类器网络。该网络无需先验知识，即可从原始传感数据中准确检测与分类故障。本研究扩展了FAILURE数据集 [1]，新增99条多模态操作记录，并为其标注了对应故障类型。FINO-Net在故障检测与分类F1分数上分别达到0.87和0.80。实验结果表明，该架构同样适用于实时场景。