We advocate the idea of the natural-language-driven(NLD) simulation to efficiently produce the object interactions between multiple objects in the virtual road scenes, for teaching and testing the autonomous driving systems that should take quick action to avoid collision with obstacles with unpredictable motions. The NLD simulation allows the brief natural-language description to control the object interactions, significantly reducing the human efforts for creating a large amount of interaction data. To facilitate the research of NLD simulation, we collect the Language-to-Interaction(L2I) benchmark dataset with 120,000 natural-language descriptions of object interactions in 6 common types of road topologies. Each description is associated with the programming code, which the graphic render can use to visually reconstruct the object interactions in the virtual scenes. As a methodology contribution, we design SimCopilot to translate the interaction descriptions to the renderable code. We use the L2I dataset to evaluate SimCopilot's abilities to control the object motions, generate complex interactions, and generalize interactions across road topologies. The L2I dataset and the evaluation results motivate the relevant research of the NLD simulation.

翻译：我们提出自然语言驱动（NLD）仿真的理念，以高效生成虚拟道路场景中多物体间的交互，用于训练和测试自动驾驶系统——这类系统需对具有不可预测运动轨迹的障碍物做出快速避让决策。NLD仿真允许通过简洁的自然语言描述控制物体交互，显著减少创建大量交互数据所需的人工投入。为促进NLD仿真研究，我们构建了语言到交互（L2I）基准数据集，包含6种常见道路拓扑结构下12万条物体交互的自然语言描述。每条描述均关联可编程代码，图形渲染器可利用这些代码在虚拟场景中视觉重建物体交互。作为方法论贡献，我们设计了SimCopilot，用于将交互描述转换为可渲染代码。我们使用L2I数据集评估SimCopilot在控制物体运动、生成复杂交互以及跨道路拓扑泛化交互方面的能力。L2I数据集及评估结果将推动NLD仿真的相关研究。