Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications.

翻译：从单目视频重建动态手-物交互对于灵巧操作数据采集以及为机器人学与虚拟现实创建真实数字孪生体至关重要。然而，现有方法面临两大瓶颈：（1）依赖神经渲染常导致在严重遮挡下产生破碎、无法直接用于仿真的几何结构；（2）依赖脆弱的运动恢复结构初始化方法，使其在真实场景视频中频繁失效。为突破这些限制，我们提出AGILE——一个将交互学习的范式从重建转向智能体生成的鲁棒框架。首先，我们采用智能体流程，通过视觉-语言模型引导生成模型合成具有高保真纹理的完整水密物体网格，其过程独立于视频遮挡。其次，我们完全绕过脆弱的运动恢复结构，提出一种鲁棒的锚点跟踪策略：利用基础模型在单帧交互起始帧初始化物体位姿，并通过生成资产与视频观测间强烈的视觉相似性进行时序传播。最后，通过接触感知优化整合语义、几何与交互稳定性约束，以增强物理合理性。在HO3D、DexYCB及真实场景视频上的大量实验表明，AGILE在全局几何精度上超越基线方法，并在现有方法常失效的挑战性序列中展现出卓越的鲁棒性。通过优先保障物理有效性，本方法产出的仿真就绪资产已通过面向机器人应用的真实-仿真重定向流程验证。