Hand-object interaction (HOI) reconstruction and synthesis are becoming central to embodied AI and AR/VR. Yet, despite rapid progress, existing HOI generation research remains fragmented across three disjoint tracks: (1) pose-only synthesis that predicts MANO trajectories without producing pixels; (2) single-image HOI generation that hallucinates appearance from masks or 2D cues but lacks dynamics; and (3) video generation methods that require both the entire pose sequence and the ground-truth first frame as inputs, preventing true sim-to-real deployment. Inspired by the philosophy of Joo et al. (2018), we think that HOI generation requires a unified engine that brings together pose, appearance, and motion within one coherent framework. Thus we introduce PAM: a Pose-Appearance-Motion Engine for controllable HOI video generation. The performance of our engine is validated by: (1) On DexYCB, we obtain an FVD of 29.13 (vs. 38.83 for InterDyn), and MPJPE of 19.37 mm (vs. 30.05 mm for CosHand), while generating higher-resolution 480x720 videos compared to 256x256 and 256x384 baselines. (2) On OAKINK2, our full multi-condition model improves FVD from 68.76 to 46.31. (3) An ablation over input conditions on DexYCB shows that combining depth, segmentation, and keypoints consistently yields the best results. (4) For a downstream hand pose estimation task using SimpleHand, augmenting training with 3,400 synthetic videos (207k frames) allows a model trained on only 50% of the real data plus our synthetic data to match the 100% real baseline.

翻译：手物交互（HOI）重建与合成正成为具身智能和AR/VR的核心技术。然而，尽管进展迅速，现有HOI生成研究仍分散在三个独立方向：（1）仅姿态合成，预测MANO轨迹但不生成像素；（2）单图像HOI生成，从掩模或2D线索中生成外观但缺乏动态性；（3）视频生成方法，需将完整姿态序列和真实首帧作为输入，阻碍了真正的仿真到现实部署。受Joo等人（2018）理念启发，我们认为HOI生成需要将姿态、外观和运动统一于一个连贯框架中的引擎。为此，我们提出PAM：一个面向可控HOI视频生成的姿态-外观-运动引擎。引擎性能通过以下实验验证：（1）在DexYCB数据集上，获得29.13的FVD（InterDyn为38.83）和19.37 mm的MPJPE（CosHand为30.05 mm），同时生成480×720高分辨率视频，而基线仅生成256×256和256×384分辨率；（2）在OAKINK2数据集上，完整多条件模型将FVD从68.76提升至46.31；（3）DexYCB上输入条件的消融实验表明，结合深度、分割和关键点一致获得最优结果；（4）使用SimpleHand进行下游手势估计任务时，用3,400个合成视频（20.7万帧）增强训练，仅用50%真实数据加合成数据训练的模型即可匹配100%真实数据基线性能。