Simulation parameter settings such as contact models and object geometry approximations are critical to training robust robotic policies capable of transferring from simulation to real-world deployment. Previous approaches typically handcraft distributions over such parameters (domain randomization), or identify parameters that best match the dynamics of the real environment (system identification). However, there is often an irreducible gap between simulation and reality: attempting to match the dynamics between simulation and reality across all states and tasks may be infeasible and may not lead to policies that perform well in reality for a specific task. Addressing this issue, we propose AdaptSim, a new task-driven adaptation framework for sim-to-real transfer that aims to optimize task performance in target (real) environments -- instead of matching dynamics between simulation and reality. First, we meta-learn an adaptation policy in simulation using reinforcement learning for adjusting the simulation parameter distribution based on the current policy's performance in a target environment. We then perform iterative real-world adaptation by inferring new simulation parameter distributions for policy training, using a small amount of real data. We perform experiments in three robotic tasks: (1) swing-up of linearized double pendulum, (2) dynamic table-top pushing of a bottle, and (3) dynamic scooping of food pieces with a spatula. Our extensive simulation and hardware experiments demonstrate AdaptSim achieving 1-3x asymptotic performance and $\sim$2x real data efficiency when adapting to different environments, compared to methods based on Sys-ID and directly training the task policy in target environments.

翻译：摘要：仿真参数设置（如接触模型和物体几何近似）对于训练能够从仿真环境成功迁移至现实部署的鲁棒机器人策略至关重要。以往方法通常人工设计这些参数的分布（域随机化），或识别最匹配真实环境动力学的参数（系统辨识）。然而，仿真与真实环境之间往往存在不可缩减的差距：试图在所有状态与任务上匹配仿真与真实环境的动力学可能不可行，且未必能生成在特定任务中表现良好的现实策略。针对此问题，我们提出AdaptSim——一种面向任务驱动的仿真到现实迁移适配框架，其目标并非匹配仿真与真实的动力学，而是优化目标（真实）环境中的任务性能。首先，我们通过强化学习在仿真中元学习一种适配策略，该策略根据当前策略在目标环境中的表现调整仿真参数分布。随后，利用少量真实数据迭代执行现实世界适配：通过推断新的仿真参数分布来训练策略。我们在三项机器人任务中开展实验：（1）线性化双摆的摆动控制，（2）动态桌面推瓶操作，（3）用铲子动态舀取食物碎片。广泛的仿真与硬件实验表明，与基于系统辨识及直接在目标环境中训练任务策略的方法相比，AdaptSim在适配不同环境时实现了1-3倍的渐进性能提升和约2倍的真实数据效率提升。