We propose NVS-HO, the first benchmark designed for novel view synthesis of handheld objects in real-world environments using only RGB inputs. Each object is recorded in two complementary RGB sequences: (1) a handheld sequence, where the object is manipulated in front of a static camera, and (2) a board sequence, where the object is fixed on a ChArUco board to provide accurate camera poses via marker detection. The goal of NVS-HO is to learn a NVS model that captures the full appearance of an object from (1), whereas (2) provides the ground-truth images used for evaluation. To establish baselines, we consider both a classical SfM pipeline and a state-of-the-art pre-trained feed-forward neural network (VGGT) as pose estimators, and train NVS models based on NeRF and Gaussian Splatting. Our experiments reveal significant performance gaps in current methods under unconstrained handheld conditions, highlighting the need for more robust approaches. NVS-HO thus offers a challenging real-world benchmark to drive progress in RGB-based novel view synthesis of handheld objects.

翻译：我们提出了NVS-HO，这是首个专为真实环境中仅使用RGB输入的手持物体新视角合成而设计的基准数据集。每个物体通过两条互补的RGB序列进行记录：(1) 手持序列，物体在静态相机前被操作；(2) 标定板序列，物体固定在ChArUco标定板上，通过标记检测提供精确的相机位姿。NVS-HO的目标是从序列(1)中学习一个能够捕捉物体完整外观的新视角合成模型，而序列(2)则提供用于评估的真实图像。为建立基线，我们同时考虑了经典的运动恢复结构流程和一种最先进的预训练前馈神经网络（VGGT）作为位姿估计器，并基于NeRF和高斯泼溅训练了新视角合成模型。我们的实验揭示了当前方法在无约束手持条件下的显著性能差距，突显了对更鲁棒方法的需求。因此，NVS-HO为基于RGB的手持物体新视角合成领域提供了一个具有挑战性的真实世界基准，以推动该方向的进展。