Large Language Models (LLMs) are trained and aligned to follow natural language instructions with only a handful of examples, and they are prompted as task-driven autonomous agents to adapt to various sources of execution environments. However, deploying agent LLMs in virtual reality (VR) has been challenging due to the lack of efficiency in online interactions and the complex manipulation categories in 3D environments. In this work, we propose Voice2Action, a framework that hierarchically analyzes customized voice signals and textual commands through action and entity extraction and divides the execution tasks into canonical interaction subsets in real-time with error prevention from environment feedback. Experiment results in an urban engineering VR environment with synthetic instruction data show that Voice2Action can perform more efficiently and accurately than approaches without optimizations.

翻译：大型语言模型（LLMs）经过训练和对齐，仅需少量示例即可遵循自然语言指令，并被提示作为任务驱动的自主智能体，适应各种执行环境。然而，在虚拟现实（VR）中部署智能体LLMs面临挑战，原因在于在线交互效率不足以及三维环境中复杂的操作类别。在本工作中，我们提出Voice2Action框架，该框架通过动作和实体提取层次化分析自定义语音信号与文本指令，将执行任务实时划分为标准交互子集，并利用环境反馈进行错误预防。在包含合成指令数据的城市工程VR环境中的实验结果表明，Voice2Action相比未优化方法能够实现更高效、更准确的执行性能。