Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.

翻译：摘要：从单目视频中重建动态手物交互对于灵巧操作数据收集以及为机器人学和虚拟现实创建逼真的数字孪生至关重要。然而，现有方法面临两大障碍：（1）依赖神经渲染通常在严重遮挡下产生碎片化、不可用于仿真计算的几何结构；（2）依赖脆弱的运动恢复结构（SfM）初始化导致在野外视频中频繁失败。为克服这些局限，我们提出AGILE——一个鲁棒的框架，将交互学习的范式从重建转向智能体生成。首先，我们采用智能体流水线：视觉语言模型（VLM）引导生成模型合成完整且水密的带高保真纹理物体网格，不受视频遮挡影响。其次，彻底绕过脆弱的SfM，我们提出鲁棒的锚点跟踪策略。通过基础模型在单个交互起始帧初始化物体姿态，并利用生成资产与视频观测之间的强视觉相似性进行时序传播。最后，基于接触的优化整合语义、几何与交互稳定性约束以强制执行物理合理性。在HO3D、DexYCB、ARCTIC及野外视频上的大量实验表明：AGILE在全局几何精度上超越基线方法，且在先前技术频繁崩溃的挑战性序列中展现出卓越的鲁棒性。通过优先保证物理有效性，我们的方法可生成经真实到仿真重定向验证的、适用于机器人应用的仿真就绪资产。项目页面：https://agile-hoi.github.io。