During teleoperation of a mobile robot, providing good operator situation awareness is a major concern as a single mistake can lead to mission failure. Camera streams are widely used for teleoperation but offer limited field-of-view. In this paper, we present a flexible framework for virtual projections to increase situation awareness based on a novel method to fuse multiple cameras mounted anywhere on the robot. Moreover, we propose a complementary approach to improve scene understanding by fusing camera images and geometric 3D Lidar data to obtain a colorized point cloud. The implementation on a compact omnidirectional camera reduces system complexity considerably and solves multiple use-cases on a much smaller footprint compared to traditional approaches such as actuated pan-tilt units. Finally, we demonstrate the generality of the approach by application to the multi-camera system of the Boston Dynamics Spot. The software implementation is available as open-source ROS packages on the project page https://tu-darmstadt-ros-pkg.github.io/omnidirectional_vision.

翻译：在移动机器人远程操作过程中，提供良好的操作员态势感知是一项关键问题，因为单一失误就可能导致任务失败。摄像头流广泛用于远程操作，但存在视野受限的缺陷。本文提出了一种基于新型多摄像头融合方法的虚拟投影灵活框架，通过将机器人上任意位置安装的多个摄像头融合，以增强态势感知。此外，我们提出了一种互补方法，通过融合摄像头图像与几何三维激光雷达数据生成彩色点云，从而提升场景理解。在紧凑型全向摄像头上的实现显著降低了系统复杂度，并以更小的物理空间解决了多个应用场景，相比传统方案（如可驱动云台单元）具有明显优势。最后，通过将其应用于波士顿动力Spot的多摄像头系统，验证了该方法的通用性。相关软件以开源ROS软件包形式发布在项目页面 https://tu-darmstadt-ros-pkg.github.io/omnidirectional_vision 上。