We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events extracted from given musical scores, to synthesize performances beyond the reference data. In addition, we present an enhanced musculoskeletal hand model that supports fine control of fingers for accurate low-level motion tracking and diverse high-level motion synthesis. We evaluate the control pipeline of our approach on a diverse piano repertoire spanning multiple musical styles and technical demands. Results demonstrate that our approach can synthesize coordinated bimanual motions with accurate key presses, and achieve the state-of-the-art performance of piano playing in physics-based dexterous control. We also show that our musculoskeletal hand model demonstrates superior biomechanical stability and tracking precision compared to the existing model, and validate that our musculoskeletal hand model and muscle-driven controller can generate physiologically plausible activation patterns that align with human electromyography (EMG) recordings.

翻译：我们提出一种数据驱动的物理模拟方法，用于实现基于肌肉驱动的灵巧控制，使肌肉骨骼手能够演奏参考数据集之外的全新钢琴曲目。该方法采用层级架构，将高频肌肉级控制与低频潜在空间协调相结合。在低层级，通过强化学习训练通用单手策略，在追踪大规模参考运动数据集轨迹的同时生成动态肌腱激活信号。随后将这些追踪策略蒸馏至变分自编码器（VAE）模型中，生成平滑且结构化的潜在空间，从而抽象化底层肌肉动力学特征。在高层级，我们训练乐曲特定策略在该潜在空间中运作，基于从给定乐谱中提取的特定目标（即音符事件）协调双手运动，进而合成超越参考数据范围的演奏效果。此外，我们提出一种增强型肌肉骨骼手模型，该模型支持手指的精细控制，以实现精确的底层运动追踪与多样化的高层运动合成。我们在涵盖多种音乐风格与技术要求的多类型钢琴曲目上对控制流水线进行评估。结果表明，该方法可合成具有精确按键的协调双手运动，并在物理模拟的灵巧控制领域达到钢琴演奏的最新性能水平。我们还证明，相较于现有模型，我们的肌肉骨骼手模型展现出更优的生物力学稳定性与追踪精度，并验证该模型与肌肉驱动控制器能够生成与人体肌电（EMG）记录相一致的生理学激活模式。