We propose a method for annotating videos of complex multi-object scenes with a globally-consistent 3D representation of the objects. We annotate each object with a CAD model from a database, and place it in the 3D coordinate frame of the scene with a 9-DoF pose transformation. Our method is semi-automatic and works on commonly-available RGB videos, without requiring a depth sensor. Many steps are performed automatically, and the tasks performed by humans are simple, well-specified, and require only limited reasoning in 3D. This makes them feasible for crowd-sourcing and has allowed us to construct a large-scale dataset by annotating real-estate videos from YouTube. Our dataset CAD-Estate offers 101k instances of 12k unique CAD models placed in the 3D representations of 20k videos. In comparison to Scan2CAD, the largest existing dataset with CAD model annotations on real scenes, CAD-Estate has 7x more instances and 4x more unique CAD models. We showcase the benefits of pre-training a Mask2CAD model on CAD-Estate for the task of automatic 3D object reconstruction and pose estimation, demonstrating that it leads to performance improvements on the popular Scan2CAD benchmark. The dataset is available at https://github.com/google-research/cad-estate.

翻译：我们提出一种方法，用于为复杂多物体场景的视频标注全局一致的物体三维表示。对于每个物体，我们从数据库匹配一个CAD模型，并通过9自由度位姿变换将其置于场景的三维坐标系中。该方法为半自动方式，适用于常见RGB视频，无需深度传感器。大多数步骤自动执行，人工操作简单明确且仅需有限的三维推理，这使得任务适合众包，从而通过标注YouTube上的房地产视频构建大规模数据集。我们的数据集CAD-Estate包含12,000个独特CAD模型的101,000个实例，分布于20,000个视频的三维表示中。与现有最大真实场景CAD模型标注数据集Scan2CAD相比，CAD-Estate的实例数量是其7倍，独特CAD模型数量是其4倍。我们展示了在CAD-Estate上预训练Mask2CAD模型用于自动三维物体重建和位姿估计任务的优势，并证明这能提升其在主流Scan2CAD基准上的性能。数据集已开源发布于https://github.com/google-research/cad-estate。