Objects manipulated by the hand (i.e., manipulanda) are particularly challenging to reconstruct from Internet videos. Not only does the hand occlude much of the object, but also the object is often only visible in a small number of image pixels. At the same time, two strong anchors emerge in this setting: (1) estimated 3D hands help disambiguate the location and scale of the object, and (2) the set of manipulanda is small relative to all possible objects. With these insights in mind, we present a scalable paradigm for hand-held object reconstruction that builds on recent breakthroughs in large language/vision models and 3D object datasets. Given a monocular RGB video, we aim to reconstruct hand-held object geometry in 3D, over time. In order to obtain the best performing single frame model, we first present MCC-Hand-Object (MCC-HO), which jointly reconstructs hand and object geometry given a single RGB image and inferred 3D hand as inputs. Subsequently, we prompt a text-to-3D generative model using GPT-4(V) to retrieve a 3D object model that matches the object in the image(s); we call this alignment Retrieval-Augmented Reconstruction (RAR). RAR provides unified object geometry across all frames, and the result is rigidly aligned with both the input images and 3D MCC-HO observations in a temporally consistent manner. Experiments demonstrate that our approach achieves state-of-the-art performance on lab and Internet image/video datasets. We make our code and models available on the project website: https://janehwu.github.io/mcc-ho

翻译：从互联网视频中重建手部操纵的物体（即可操纵物）具有特殊挑战性。不仅手部会严重遮挡物体，而且物体通常仅在少量图像像素中可见。与此同时，该场景中存在两个强有力的锚定依据：（1）估计的3D手部有助于消除物体位置与尺度的歧义；（2）可操纵物的集合相对于所有可能物体而言规模较小。基于这些认知，我们提出了一种可扩展的手持物体重建范式，该范式建立在大型语言/视觉模型与3D物体数据集的最新突破之上。给定单目RGB视频，我们的目标是在时间维度上重建手持物体的三维几何形状。为获得最佳性能的单帧模型，我们首先提出MCC-Hand-Object（MCC-HO），该模型以单张RGB图像和推断的3D手部作为输入，联合重建手部与物体的几何结构。随后，我们利用GPT-4(V)提示文本到3D生成模型，以检索与图像中物体相匹配的3D物体模型；我们将此对齐过程称为检索增强重建（RAR）。RAR为所有帧提供统一的物体几何表示，并以时间一致的方式与输入图像及3D MCC-HO观测结果进行刚性对齐。实验表明，我们的方法在实验室及互联网图像/视频数据集上实现了最先进的性能。相关代码与模型已在项目网站发布：https://janehwu.github.io/mcc-ho