Robots have been increasingly better at doing tasks for humans by learning from their feedback, but still often suffer from model misalignment due to missing or incorrectly learned features. When the features the robot needs to learn to perform its task are missing or do not generalize well to new settings, the robot will not be able to learn the task the human wants and, even worse, may learn a completely different and undesired behavior. Prior work shows how the robot can detect when its representation is missing some feature and can, thus, ask the human to be taught about the new feature; however, these works do not differentiate between features that are completely missing and those that exist but do not generalize to new environments. In the latter case, the robot would detect misalignment and simply learn a new feature, leading to an arbitrarily growing feature representation that can, in turn, lead to spurious correlations and incorrect learning down the line. In this work, we propose separating the two sources of misalignment: we propose a framework for determining whether a feature the robot needs is incorrectly learned and does not generalize to new environment setups vs. is entirely missing from the robot's representation. Once we detect the source of error, we show how the human can initiate the realignment process for the model: if the feature is missing, we follow prior work for learning new features; however, if the feature exists but does not generalize, we use data augmentation to expand its training and, thus, complete the correction. We demonstrate the proposed approach in experiments with a simulated 7DoF robot manipulator and physical human corrections.

翻译：机器人通过从人类反馈中学习，在执行任务方面已日益擅长，但仍常因缺失或错误学习的特征而面临模型对齐问题。当机器人完成任务所需的特征缺失或无法良好泛化至新场景时，它将无法学习人类期望的任务，更甚者可能习得完全不同的非预期行为。已有研究展示了机器人如何检测其表示中是否存在特征缺失，并据此请求人类教授新特征；然而，这些工作并未区分完全缺失的特征与虽存在却无法泛化至新环境的特征。在后一种情况下，机器人会检测到对齐错误并简单学习新特征，导致特征表示任意增长，进而引发虚假关联与后续错误学习。本研究提出将两种对齐错误来源进行区分：我们构建一个框架，用于判定机器人所需特征属于错误学习且无法泛化至新环境场景，还是完全缺失于机器人表示中。一旦定位错误来源，我们展示了人类如何启动模型重新对齐过程：若特征缺失，则沿用已有研究中的新特征学习方法；若特征存在但无法泛化，则通过数据增强扩展其训练以完成修正。我们通过在模拟七自由度机器人操纵器与真实人类修正实验中的演示验证了所提方法。