Inferring the structure of 3D scenes from 2D observations is a fundamental challenge in computer vision. Recently popularized approaches based on neural scene representations have achieved tremendous impact and have been applied across a variety of applications. One of the major remaining challenges in this space is training a single model which can provide latent representations which effectively generalize beyond a single scene. Scene Representation Transformer (SRT) has shown promise in this direction, but scaling it to a larger set of diverse scenes is challenging and necessitates accurately posed ground truth data. To address this problem, we propose RUST (Really Unposed Scene representation Transformer), a pose-free approach to novel view synthesis trained on RGB images alone. Our main insight is that one can train a Pose Encoder that peeks at the target image and learns a latent pose embedding which is used by the decoder for view synthesis. We perform an empirical investigation into the learned latent pose structure and show that it allows meaningful test-time camera transformations and accurate explicit pose readouts. Perhaps surprisingly, RUST achieves similar quality as methods which have access to perfect camera pose, thereby unlocking the potential for large-scale training of amortized neural scene representations.

翻译：从二维观测推断三维场景结构是计算机视觉中的一项基础挑战。近期流行的基于神经场景表征的方法已产生巨大影响，并被应用于多种场景。该领域尚存的主要难题之一是训练一个能提供有效泛化至单场景之外的隐式表征的单一模型。场景表征变换器（SRT）在此方向上展现出潜力，但将其扩展到更大规模、更多样化的场景集仍具挑战性，且需要精确的位姿标注数据。为解决此问题，我们提出RUST（真实未定位场景表征变换器），一种仅基于RGB图像训练、无需位姿信息的新型视图合成方法。我们的核心洞察在于：可训练一个观察目标图像并学习隐式位姿嵌入的位姿编码器，该嵌入由解码器用于视图合成。我们对所学隐式位姿结构进行实证研究，表明其支持有意义的测试阶段相机变换和准确的显式位姿读取。令人意外的是，RUST可实现与具有完美相机位姿方法相当的质量，从而为放大训练摊销神经场景表征释放了潜力。