Accurate and safe robotic manipulation under dynamic and visually occluded conditions remains a core challenge in real-world deployment. We introduce SyncTwin, a novel digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware robotic manipulation in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states via point cloud segmentation updates and aligning them through colored-ICP registration. The synchronized twin enables motion planners to compute collision-free and dynamically feasible trajectories in simulation, which are safely executed on the real robot through a closed real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves manipulation performance and motion safety, demonstrating the effectiveness of digital twin synchronization for real-world robotic execution. The video demos and code can be found on the project website: https://sync-twin.github.io/.
翻译:在动态与视觉遮挡条件下实现精确且安全的机器人操作,仍是实际部署中的核心挑战。本文提出SyncTwin,一种新颖的数字孪生框架,它集成了快速三维场景重建与实-仿同步功能,旨在为此类环境提供鲁棒且具备安全意识的机器人操作。在离线阶段,我们采用VGGT从RGB图像快速重建物体级三维资产,形成可复用的几何库。在执行过程中,SyncTwin通过点云分割更新跟踪真实世界物体状态,并借助彩色-ICP配准进行对齐,从而持续同步数字孪生。同步后的孪生体使得运动规划器能够在仿真中计算无碰撞且动态可行的轨迹,并通过闭环的实-仿-实回路在真实机器人上安全执行。在动态与遮挡场景中的实验表明,SyncTwin提升了操作性能与运动安全性,验证了数字孪生同步对于真实世界机器人执行的有效性。视频演示与代码可在项目网站查看:https://sync-twin.github.io/。