Developing embodied agents in simulation has been a key research topic in recent years. Exciting new tasks, algorithms, and benchmarks have been developed in various simulators. However, most of them assume deaf agents in silent environments, while we humans perceive the world with multiple senses. We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear. Sonicverse models realistic continuous audio rendering in 3D environments in real-time. Together with a new audio-visual VR interface that allows humans to interact with agents with audio, Sonicverse enables a series of embodied AI tasks that need audio-visual perception. For semantic audio-visual navigation in particular, we also propose a new multi-task learning model that achieves state-of-the-art performance. In addition, we demonstrate Sonicverse's realism via sim-to-real transfer, which has not been achieved by other simulators: an agent trained in Sonicverse can successfully perform audio-visual navigation in real-world environments. Sonicverse is available at: https://github.com/StanfordVL/Sonicverse.

翻译：近年来，在仿真环境中开发具身智能体已成为关键研究课题。各类仿真器中涌现出激动人心的新任务、算法与基准测试，但多数默认智能体处于静默环境——而人类通过多重感官感知世界。我们提出Sonicverse，一个集成视听仿真的多感官仿真平台，用于训练既能视觉感知又能听觉感知的家庭智能体。Sonicverse在三维环境中实时建模连续音频渲染，结合允许人类通过音频与智能体交互的新型视听VR接口，使一系列需要视听感知的具身AI任务成为可能。针对语义视听导航任务，我们进一步提出新型多任务学习模型，实现了当前最优性能。此外，通过其他仿真器尚未实现的"仿真到现实迁移"，我们验证了Sonicverse的真实性：在Sonicverse中训练的智能体可成功在真实世界环境中执行视听导航。Sonicverse开源地址：https://github.com/StanfordVL/Sonicverse。