When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of the type, or quality, of human feedback. However, in many settings, giving one type of feedback (e.g. a demonstration) may be much more difficult than a different type of feedback (e.g. answering a comparison query). Thus, we expect to see more or less noise depending on the type of human feedback. In this work, we advocate that grounding the rationality coefficient in real data for each feedback type, rather than assuming a default value, has a significant positive effect on reward learning. We test this in both simulated experiments and in a user study with real human feedback. We find that overestimating human rationality can have dire effects on reward learning accuracy and regret. We also find that fitting the rationality coefficient to human data enables better reward learning, even when the human deviates significantly from the noisy-rational choice model due to systematic biases. Further, we find that the rationality level affects the informativeness of each feedback type: surprisingly, demonstrations are not always the most informative -- when the human acts very suboptimally, comparisons actually become more informative, even when the rationality level is the same for both. Ultimately, our results emphasize the importance and advantage of paying attention to the assumed human-rationality level, especially when agents actively learn from multiple types of human feedback.

翻译：在从人类行为（无论是演示、比较、物理纠正还是紧急停止）中推断奖励函数时，将人类建模为做出带噪声的理性选择已被证明是有效的，其中“理性系数”用于捕捉我们在人类行为中预期观察到的噪声或熵的程度。先前的工作通常将理性水平设置为恒定值，而不考虑人类反馈的类型或质量。然而，在许多情境中，提供一种类型的反馈（例如演示）可能比另一种类型的反馈（例如回答比较查询）困难得多。因此，我们预期根据人类反馈的类型不同，会观察到或多或少的噪声。在这项工作中，我们主张根据每种反馈类型的真实数据来校准理性系数，而不是采用默认值，这对奖励学习具有显著的正面影响。我们在模拟实验和包含真实人类反馈的用户研究中对此进行了测试。我们发现，高估人类理性程度会对奖励学习的准确性和遗憾值产生严重后果。我们还发现，即使人类因系统性偏差而显著偏离带噪声的理性选择模型，将理性系数拟合到人类数据仍能实现更好的奖励学习。此外，我们发现理性水平会影响每种反馈类型的信息量：令人惊讶的是，演示并非总是信息量最大的——当人类行为高度次优时，比较反而变得更具信息性，即使两种反馈类型的理性水平相同。最终，我们的结果强调了关注假定的理性水平的重要性和优势，尤其是在智能体主动从多种类型的人类反馈中进行学习时。