Metaverse technologies demand accurate, real-time, and immersive modeling on consumer-grade hardware for both non-human perception (e.g., drone/robot/autonomous car navigation) and immersive technologies like AR/VR, requiring both structural accuracy and photorealism. However, there exists a knowledge gap in how to apply geometric reconstruction and photorealism modeling (novel view synthesis) in a unified framework. To address this gap and promote the development of robust and immersive modeling and rendering with consumer-grade devices, we propose a real-world Multi-Sensor Hybrid Room Dataset (MuSHRoom). Our dataset presents exciting challenges and requires state-of-the-art methods to be cost-effective, robust to noisy data and devices, and can jointly learn 3D reconstruction and novel view synthesis instead of treating them as separate tasks, making them ideal for real-world applications. We benchmark several famous pipelines on our dataset for joint 3D mesh reconstruction and novel view synthesis. Our dataset and benchmark show great potential in promoting the improvements for fusing 3D reconstruction and high-quality rendering in a robust and computationally efficient end-to-end fashion. The dataset and code are available at the project website: https://xuqianren.github.io/publications/MuSHRoom/.

翻译：元宇宙技术需要在消费级硬件上实现精确、实时且沉浸式的建模，以满足非人类感知（如无人机/机器人/自动驾驶汽车导航）以及AR/VR等沉浸式技术的需求，这同时要求结构精度与照片级真实感。然而，如何在统一框架中应用几何重建与照片级真实感建模（新视角合成）仍存在知识空白。为填补这一空白并推动基于消费级设备的鲁棒沉浸式建模与渲染技术的发展，我们提出了一个真实世界的多传感器混合房间数据集（MuSHRoom）。本数据集提出了引人注目的挑战，要求前沿方法具备成本效益、对噪声数据与设备具有鲁棒性，并能联合学习三维重建与新视角合成而非将其视为独立任务，从而使其非常适合实际应用。我们在数据集上对多个知名流程进行了联合三维网格重建与新视角合成的基准测试。我们的数据集与基准测试在推动以鲁棒且计算高效端到端方式融合三维重建与高质量渲染的改进方面展现出巨大潜力。数据集与代码可在项目网站获取：https://xuqianren.github.io/publications/MuSHRoom/。