The current transformation towards smart manufacturing has led to a growing demand for human-robot collaboration (HRC) in the manufacturing process. Perceiving and understanding the human co-worker's behaviour introduces challenges for collaborative robots to efficiently and effectively perform tasks in unstructured and dynamic environments. Integrating recent data-driven machine vision capabilities into HRC systems is a logical next step in addressing these challenges. However, in these cases, off-the-shelf components struggle due to generalisation limitations. Real-world evaluation is required in order to fully appreciate the maturity and robustness of these approaches. Furthermore, understanding the pure-vision aspects is a crucial first step before combining multiple modalities in order to understand the limitations. In this paper, we propose GoferBot, a novel vision-based semantic HRC system for a real-world assembly task. It is composed of a visual servoing module that reaches and grasps assembly parts in an unstructured multi-instance and dynamic environment, an action recognition module that performs human action prediction for implicit communication, and a visual handover module that uses the perceptual understanding of human behaviour to produce an intuitive and efficient collaborative assembly experience. GoferBot is a novel assembly system that seamlessly integrates all sub-modules by utilising implicit semantic information purely from visual perception.

翻译：当前向智能制造的转型导致制造过程中对人机协作（HRC）的需求日益增长。感知和理解人类同事的行为为协作机器人在非结构化动态环境中高效、有效地执行任务带来了挑战。将最新的数据驱动机器视觉能力集成到HRC系统中，是应对这些挑战的合理下一步。然而，在这些情况下，现成组件因泛化限制而难以胜任。要全面评估这些方法的成熟度和鲁棒性，需要进行实际环境评估。此外，在融合多种模态以理解其局限性之前，理解纯视觉方面是关键的第一步。本文提出了GoferBot——一种面向真实世界装配任务的基于视觉语义的新型HRC系统。该系统包含：视觉伺服模块，可在非结构化多实例动态环境中抓取装配部件；动作识别模块，用于执行人类动作预测以实现隐式通信；以及视觉交接模块，利用对人类行为的感知理解来提供直观高效的协同装配体验。GoferBot是一种新型装配系统，通过仅从视觉感知中利用隐式语义信息，无缝集成了所有子模块。