Learning real-world dynamics from visual observations is crucial for various domains. A common strategy is to calibrate simulators by estimating physical parameters, yet accuracy is ultimately bounded by the underlying physical models, which often assume materials are homogeneous and isotropic. Even if reasonable, real-world objects typically exhibit mild anisotropy and heterogeneity. After the near-isotropic backbone is well calibrated, these residual effects become the key bottleneck for further closing the real-to-sim gap. Although neural networks can fit dynamics end-to-end, such black-box modeling discards strong physical priors, leading to poor data efficiency and overfitting. Therefore, we propose MoSA, a motion-constrained stress adaptation framework that targets these residual effects to further improve real-to-sim dynamics learning. MoSA uses an isotropic model as a physics prior and learns residual stress operators to capture mild anisotropy and heterogeneity. It progressively adapts stresses via microplane-constrained redistribution in a physics-informed cascaded network. We further impose motion constraints by supervising temporal and spatial derivatives of the deformation field. Experimentally, our learned dynamics achieves superior accuracy, generalization, and robustness, while learning physically meaningful residual anisotropy. Finally, we validate MoSA in a robot manipulation setting, showing that better real-to-sim dynamics modeling translates into more reliable sim-to-real transfer. Project Page is available at https://mercerai.github.io/MoSA/.

翻译：从视觉观测学习真实世界动力学对于多个领域至关重要。常见策略是通过估计物理参数来校准仿真器，但其精度最终受限于底层物理模型——这些模型通常假设材料是均匀且各向同性的。即使假设合理，真实物体也往往呈现轻微的各向异性和非均匀性。在近各向同性主干被充分校准后，这些残余效应成为进一步缩小真实-仿真差距的关键瓶颈。尽管神经网络能够端到端拟合动力学，但这种黑盒建模丢弃了强物理先验，导致数据效率低下和过拟合。为此，我们提出MoSA——一种面向残余效应的运动约束应力自适应框架，旨在进一步改进真实-仿真动力学学习。MoSA以各向同性模型作为物理先验，并学习残余应力算子以捕捉轻微各向异性和非均匀性。它通过物理信息级联网络中的微平面约束重分配来渐进式适应应力。我们进一步通过监督变形场的时空导数施加运动约束。实验表明，我们习得的动力学实现了卓越的精度、泛化性和鲁棒性，同时学习到具有物理意义的残余各向异性。最后，我们在机器人操控场景中验证了MoSA，证明更优的真实-仿真动力学建模能转化为更可靠的仿真-真实迁移。项目页面见https://mercerai.github.io/MoSA/。