Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications.

翻译：从单视角视频中重建动态手物交互对灵巧操作数据采集及为机器人与虚拟现实创建逼真数字孪生至关重要。然而，现有方法面临两大障碍：(1) 依赖神经渲染常导致严重遮挡下产生碎片化、不可用于仿真的几何结构；(2) 依赖脆弱的运动恢复结构（SfM）初始化在野外视频中频繁失效。为克服这些局限，我们提出AGILE——一个将交互学习范式从重建转向智能体生成的鲁棒框架。首先，我们采用智能体流水线：视觉语言模型（VLM）引导生成模型合成具有高保真纹理的完整水密物体网格，不受视频遮挡影响。其次，完全规避脆弱的SfM技术，我们提出鲁棒的锚点-追踪策略：利用基础模型在单个交互起始帧初始化物体位姿，并借助生成资产与视频观测之间的强视觉相似性实现时间传播。最终，结合语义、几何与交互稳定性约束的接触感知优化，确保物理合理性。在HO3D、DexYCB及野外视频上的大量实验表明，AGILE在全局几何精度上超越基线方法，并在现有技术频繁崩溃的挑战性序列中展现出卓越鲁棒性。通过优先保障物理有效性，我们的方法生成的资产可直接用于机器人领域的实到仿映射。