The learn-from-observation (LfO) paradigm is a human-inspired mode for a robot to learn to perform a task simply by watching it being performed. LfO can facilitate robot integration on factory floors by minimizing disruption and reducing tedious programming. A key component of the LfO pipeline is a transformation of the depth camera frames to the corresponding task state and action pairs, which are then relayed to learning techniques such as imitation or inverse reinforcement learning for understanding the task parameters. While several existing computer vision models analyze videos for activity recognition, SA-Net specifically targets robotic LfO from RGB-D data. However, SA-Net and many other models analyze frame data captured from a single viewpoint. Their analysis is therefore highly sensitive to occlusions of the observed task, which are frequent in deployments. An obvious way of reducing occlusions is to simultaneously observe the task from multiple viewpoints and synchronously fuse the multiple streams in the model. Toward this, we present multi-view SA-Net, which generalizes the SA-Net model to allow the perception of multiple viewpoints of the task activity, integrate them, and better recognize the state and action in each frame. Performance evaluations on two distinct domains establish that MVSA-Net recognizes the state-action pairs under occlusion more accurately compared to single-view MVSA-Net and other baselines. Our ablation studies further evaluate its performance under different ambient conditions and establish the contribution of the architecture components. As such, MVSA-Net offers a significantly more robust and deployable state-action trajectory generation compared to previous methods.

翻译：学习-通过-观察（LfO）范式是一种受人类启发的机器人学习模式，使其仅通过观察任务执行过程即可学会执行任务。LfO能够通过减少干扰和繁琐编程，促进机器人集成到工厂车间。LfO流程的关键组成部分是将深度相机帧转换为相应的任务状态-动作对，随后这些数据被传递给模仿学习或逆强化学习等技术以理解任务参数。尽管现有的多种计算机视觉模型通过分析视频进行活动识别，但SA-Net专为基于RGB-D数据的机器人LfO设计。然而，SA-Net及许多其他模型仅分析从单一视角捕获的帧数据，因此其分析对任务观测中的遮挡高度敏感，而遮挡在部署中频繁出现。减少遮挡的直观方法是从多个视角同步观测任务，并在模型中融合多路数据流。为此，我们提出多视角SA-Net（MVSA-Net），将SA-Net模型泛化为可感知任务活动的多视角信息、融合这些信息，并更精准地识别每帧中的状态和动作。在两个不同领域的性能评估表明，与单视角SA-Net及其他基线方法相比，MVSA-Net在遮挡条件下能更准确地识别状态-动作对。我们的消融研究进一步评估了其在不同环境条件下的性能，并验证了网络架构各组件的贡献。因此，相较于先前方法，MVSA-Net能够实现显著更鲁棒且可部署的状态-动作轨迹生成。