Robots must be able to understand their surroundings to perform complex tasks in challenging environments and many of these complex tasks require estimates of physical properties such as friction or weight. Estimating such properties using learning is challenging due to the large amounts of labelled data required for training and the difficulty of updating these learned models online at run time. To overcome these challenges, this paper introduces a novel, multi-modal approach for representing semantic predictions and physical property estimates jointly in a probabilistic manner. By using conjugate pairs, the proposed method enables closed-form Bayesian updates given visual and tactile measurements without requiring additional training data. The efficacy of the proposed algorithm is demonstrated through several hardware experiments. In particular, this paper illustrates that by conditioning semantic classifications on physical properties, the proposed method quantitatively outperforms state-of-the-art semantic classification methods that rely on vision alone. To further illustrate its utility, the proposed method is used in several applications including to represent affordance-based properties probabilistically and a challenging terrain traversal task using a legged robot. In the latter task, the proposed method represents the coefficient of friction of the terrain probabilistically, which enables the use of an on-line risk-aware planner that switches the legged robot from a dynamic gait to a static, stable gait when the expected value of the coefficient of friction falls below a given threshold. Videos of these case studies are presented in the multimedia attachment. The proposed framework includes an open-source C++ and ROS interface.

翻译：机器人必须在复杂环境中理解周围环境以执行高级任务，而其中许多任务需要估算摩擦系数或重量等物理属性。由于训练所需的大量标注数据以及运行时在线更新学习模型的困难，基于学习的方法在估算这些属性时面临挑战。为克服这些难题，本文提出一种新颖的多模态方法，以概率形式联合表示语义预测与物理属性估计。通过利用共轭分布对，所提方法在无需额外训练数据的情况下，即可根据视觉与触觉测量实现闭式贝叶斯更新。通过多项硬件实验验证了该算法的有效性。特别地，本文展示了通过将语义分类与物理属性关联，所提方法在定量上优于仅依赖视觉的现有最优语义分类方法。为进一步说明其实用性，所提方法被应用于多个场景，包括基于功能属性的概率化表示以及使用腿式机器人的复杂地形穿越任务。在后者中，所提方法以概率形式表示地形摩擦系数，从而使在线风险感知规划器能够在摩擦系数期望值低于设定阈值时，将机器人从动态步态切换为静态稳定步态。这些案例研究的视频见多媒体附件。所提框架包含开源的C++和ROS接口。