We present Fillerbuster, a unified model that completes unknown regions of a 3D scene with a multi-view latent diffusion transformer. Casual captures are often sparse and miss surrounding content behind objects or above the scene. Existing methods are not suitable for this challenge as they focus on making known pixels look good with sparse-view priors, or on creating missing sides of objects from just one or two photos. In reality, we often have hundreds of input frames and want to complete areas that are missing and unobserved from the input frames. Our solution is to train a generative model that can consume a large context of input frames while generating unknown target views and recovering image poses when camera parameters are unknown. We show results where we complete partial captures on two existing datasets. We also present an uncalibrated scene completion task where our unified model predicts both poses and creates new content. We open-source our framework for integration into popular reconstruction platforms like Nerfstudio or Gsplat. We present a flexible, unified inpainting framework to predict many images and poses together, where all inputs are jointly inpainted, and it could be extended to predict more modalities such as depth.

翻译：我们提出了Fillerbuster，一种通过多视角潜在扩散Transformer补全三维场景未知区域的统一模型。随意采集的数据通常较为稀疏，会缺失物体后方或场景上方的周边内容。现有方法不适用于这一挑战，因为它们侧重于利用稀疏视角先验使已知像素呈现良好效果，或仅从一两张照片中创建物体的缺失侧面。实际上，我们通常拥有数百张输入帧，并希望补全输入帧中缺失且未被观测到的区域。我们的解决方案是训练一个生成模型，该模型能够处理大量输入帧的上下文信息，同时生成未知的目标视角，并在相机参数未知时恢复图像位姿。我们在两个现有数据集上展示了补全部分采集数据的结果。我们还提出了一个未标定场景补全任务，其中我们的统一模型同时预测位姿并创建新内容。我们开源了该框架，以便集成到Nerfstudio或Gsplat等主流重建平台中。我们提出了一种灵活统一的修复框架，可同时预测多张图像与位姿，其中所有输入均进行联合修复，并可扩展至预测深度等其他模态。