We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing a 3D object from a few unposed images even with little visual overlap, while simultaneously estimating the relative camera poses in ~1.3 seconds on a single A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention blocks to exchange information between 3D object tokens and 2D image tokens; we predict a coarse point cloud for each view, and then use a differentiable Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge amount of multi-view posed data of ~1M objects, PF-LRM shows strong cross-dataset generalization ability, and outperforms baseline methods by a large margin in terms of pose prediction accuracy and 3D reconstruction quality on various unseen evaluation datasets. We also demonstrate our model's applicability in downstream text/image-to-3D task with fast feed-forward inference. Our project website is at: https://totoro97.github.io/pf-lrm .

翻译：我们提出了一种无位姿的大规模重建模型（PF-LRM），该模型能够从少量未标定姿态的图像（即使视觉重叠度很低）中重建三维物体，同时在大约1.3秒内（单块A100 GPU）估计出相对相机位姿。PF-LRM是一种高度可扩展的方法，利用自注意力模块在三维物体标记与二维图像标记之间交换信息；我们为每个视角预测一个粗点云，然后通过可微的透视n点算法（PnP）求解器获取相机位姿。当在大规模（约100万个物体）多视角标定姿态数据上训练时，PF-LRM展现出强大的跨数据集泛化能力，并在多个未见过的评估数据集上，在姿态预测精度与三维重建质量方面大幅优于基线方法。我们还展示了模型在下游文本/图像到三维任务中的适用性，支持快速的前馈推理。我们的项目网站地址为：https://totoro97.github.io/pf-lrm。