Developing a unified multi-task foundation model has become a critical challenge in computer vision research. In the current field of 3D computer vision, most datasets only focus on single task, which complicates the concurrent training requirements of various downstream tasks. In this paper, we introduce VEnvision3D, a large 3D synthetic perception dataset for multi-task learning, including depth completion, segmentation, upsampling, place recognition, and 3D reconstruction. Since the data for each task is collected in the same environmental domain, sub-tasks are inherently aligned in terms of the utilized data. Therefore, such a unique attribute can assist in exploring the potential for the multi-task model and even the foundation model without separate training methods. Meanwhile, capitalizing on the advantage of virtual environments being freely editable, we implement some novel settings such as simulating temporal changes in the environment and sampling point clouds on model surfaces. These characteristics enable us to present several new benchmarks. We also perform extensive studies on multi-task end-to-end models, revealing new observations, challenges, and opportunities for future research. Our dataset and code will be open-sourced upon acceptance.

翻译：构建统一的通用多任务基础模型已成为计算机视觉研究中的关键挑战。在当前3D计算机视觉领域，大多数数据集仅聚焦于单一任务，这使得各下游任务的并发训练需求变得复杂。本文提出VEnvision3D——一个用于多任务学习的大规模3D合成感知数据集，涵盖深度补全、分割、上采样、位置识别与三维重建等任务。由于各任务数据采集于相同的环境域，子任务在所用数据上天然对齐。因此，这一独特属性有助于在不依赖独立训练方法的情况下探索多任务模型乃至基础模型的潜在能力。同时，借助虚拟环境可自由编辑的优势，我们实现了若干新颖设置，例如模拟环境的时间变化以及在模型表面采样点云。这些特性使我们能够提出多个新基准。此外，我们对多任务端到端模型进行了广泛研究，揭示了未来研究的新发现、挑战与机遇。我们的数据集与代码将在论文接收后开源。