Towards the grand challenge of achieving human-level manipulation in robots, playing piano is a compelling testbed that requires strategic, precise, and flowing movements. Over the years, several works demonstrated hand-designed controllers on real world piano playing, while other works evaluated robot learning approaches on simulated piano playing. In this work, we develop the first piano playing robotic system that makes use of learning approaches while also being deployed on a real world dexterous robot. Specifically, we use a Sim2Real2Sim approach where we iteratively alternate between training policies in simulation, deploying the policies in the real world, and use the collected real world data to update the parameters of the simulator. Using this approach we demonstrate that the robot can learn to play several piano pieces (including Are You Sleeping, Happy Birthday, Ode To Joy, and Twinkle Twinkle Little Star) in the real world accurately, reaching an average F1-score of 0.881. By providing this proof-of-concept, we want to encourage the community to adopt piano playing as a compelling benchmark towards human-level manipulation in the real world. We open-source our code and show additional videos at www.lasr.org/research/learning-to-play-piano .

翻译：为了实现机器人人类级操控这一宏大挑战，钢琴弹奏是一个极具说服力的试验平台，它要求动作具备策略性、精准性与流畅性。多年来，多项研究展示了在真实钢琴上使用手工设计控制器的成果，而其他工作则在仿真环境中评估了机器人学习方法。在本研究中，我们开发了首个结合学习方法并部署于真实灵巧机器人的钢琴弹奏系统。具体而言，我们采用Sim2Real2Sim方法，通过迭代交替进行仿真策略训练、真实世界策略部署，并利用采集的真实数据更新仿真器参数。通过该方法，我们证明了机器人能在真实世界中准确学会弹奏多首钢琴曲目（包括《你睡着了吗》《生日快乐》《欢乐颂》以及《小星星》），平均F1分数达到0.881。通过这一概念验证，我们希望鼓励学界将钢琴弹奏作为迈向真实世界人类级操控的重要标杆。我们已在www.lasr.org/research/learning-to-play-piano开源代码并展示更多视频。