This work highlights a critical shortcoming in text-based Large Language Models (LLMs) used for human-robot interaction, demonstrating that text alone as a conversation modality falls short in such applications. While LLMs excel in processing text in these human conversations, they struggle with the nuances of verbal instructions in scenarios like social navigation, where ambiguity and uncertainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not involve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present "Beyond Text"; an approach that improves LLM decision-making by integrating audio transcription along with a subsection of these features, which focus on the affect and more relevant in human-robot conversations. This approach not only achieves a 70.26% winning rate, outperforming existing LLMs by 48.30%, but also enhances robustness against token manipulation adversarial attacks, highlighted by a 22.44% less decrease ratio than the text-only language model in winning rate. "Beyond Text" marks an advancement in social robot navigation and broader Human-Robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.

翻译：这项工作揭示了基于文本的大语言模型在人机交互中的一个关键缺陷，证明仅依靠文本作为对话模态在此类应用中存在不足。尽管LLM在处理人类对话中的文本时表现出色，但在社交导航等场景中，它们难以应对口头指令的细微差别——其中模糊性和不确定性会削弱用户对机器人及其他AI系统的信任。为弥补这一缺陷，我们主张超越文本范畴，额外关注音频响应的副语言特征。这些特征涉及口语交流中不包含字面语义（词汇内容）的方面，通过表达方式传递含义与细微差别。我们提出"Beyond Text"方法：通过整合音频转录及其专注于情感层面的相关副语言特征（这类特征在人机对话中更具相关性），提升LLM的决策能力。该方法不仅实现了70.26%的胜率（较现有LLM提升48.30%），还将对抗性令牌操纵攻击造成的胜率下降幅度较纯文本语言模型降低22.44%。"Beyond Text"标志着社交机器人导航及更广泛人机交互领域的进步，成功将基于文本的指导与蕴含人类音频信息的语言模型无缝融合。