We advocate the idea of the natural-language-driven(NLD) simulation to efficiently produce the object interactions between multiple objects in the virtual road scenes, for teaching and testing the autonomous driving systems that should take quick action to avoid collision with obstacles with unpredictable motions. The NLD simulation allows the brief natural-language description to control the object interactions, significantly reducing the human efforts for creating a large amount of interaction data. To facilitate the research of NLD simulation, we collect the Language-to-Interaction(L2I) benchmark dataset with 120,000 natural-language descriptions of object interactions in 6 common types of road topologies. Each description is associated with the programming code, which the graphic render can use to visually reconstruct the object interactions in the virtual scenes. As a methodology contribution, we design SimCopilot to translate the interaction descriptions to the renderable code. We use the L2I dataset to evaluate SimCopilot's abilities to control the object motions, generate complex interactions, and generalize interactions across road topologies. The L2I dataset and the evaluation results motivate the relevant research of the NLD simulation.

翻译：我们提出自然语言驱动（NLD）仿真的理念，以高效生成虚拟道路场景中多物体间的交互行为，用于教学和测试自动驾驶系统——此类系统需快速采取行动以避免与运动不可预测的障碍物发生碰撞。NLD仿真允许通过简短的自然语言描述来控制物体交互，显著减少了创建大量交互数据所需的人力投入。为促进NLD仿真研究，我们构建了语言到交互（L2I）基准数据集，其中包含12万条针对6种常见道路拓扑结构的物体交互自然语言描述。每条描述均对应一段编程代码，图形渲染器可利用这些代码在虚拟场景中重建物体交互的可视化效果。作为方法论贡献，我们设计了SimCopilot将交互描述转化为可执行渲染代码。我们使用L2I数据集评估了SimCopilot在控制物体运动、生成复杂交互以及跨道路拓扑结构泛化交互方面的能力。L2I数据集及评估结果将推动NLD仿真的相关研究。