Simulation parameter settings such as contact models and object geometry approximations are critical to training robust robotic policies capable of transferring from simulation to real-world deployment. Previous approaches typically handcraft distributions over such parameters (domain randomization), or identify parameters that best match the dynamics of the real environment (system identification). However, there is often an irreducible gap between simulation and reality: attempting to match the dynamics between simulation and reality across all states and tasks may be infeasible and may not lead to policies that perform well in reality for a specific task. Addressing this issue, we propose AdaptSim, a new task-driven adaptation framework for sim-to-real transfer that aims to optimize task performance in target (real) environments -- instead of matching dynamics between simulation and reality. First, we meta-learn an adaptation policy in simulation using reinforcement learning for adjusting the simulation parameter distribution based on the current policy's performance in a target environment. We then perform iterative real-world adaptation by inferring new simulation parameter distributions for policy training, using a small amount of real data. We perform experiments in three robotic tasks: (1) swing-up of linearized double pendulum, (2) dynamic table-top pushing of a bottle, and (3) dynamic scooping of food pieces with a spatula. Our extensive simulation and hardware experiments demonstrate AdaptSim achieving 1-3x asymptotic performance and $\sim$2x real data efficiency when adapting to different environments, compared to methods based on Sys-ID and directly training the task policy in target environments. Website: https://irom-lab.github.io/AdaptSim/

翻译：仿真参数设置（如接触模型与物体几何近似）对于训练能够从仿真迁移至现实部署的鲁棒机器人策略至关重要。以往方法通常手动构建此类参数的分布（域随机化），或识别与真实环境动力学最匹配的参数（系统辨识）。然而，仿真与真实之间往往存在不可缩减的鸿沟：试图在所有状态和任务上匹配仿真与现实的动力学可能不可行，且未必能产生在现实中针对特定任务表现优异的策略。针对此问题，我们提出AdaptSim——一种面向Sim-to-Real迁移的任务驱动适配框架，旨在优化目标（真实）环境中的任务性能，而非匹配仿真与现实的动力学。首先，我们在仿真中利用强化学习元学习一个适配策略，根据当前策略在目标环境中的表现调整仿真参数分布。随后，通过少量真实数据推断新的仿真参数分布用于策略训练，进行迭代式真实环境适配。我们在三项机器人任务上开展实验：（1）线性化双摆的摆起、（2）桌面动态推瓶子、（3）用锅铲动态盛取食物块。大规模仿真与硬件实验表明，与基于系统辨识以及直接在目标环境中训练任务策略的方法相比，AdaptSim在不同环境适配中实现了1-3倍渐近性能提升和约2倍真实数据效率提升。网站：https://irom-lab.github.io/AdaptSim/