Developing a unified multi-task foundation model has become a critical challenge in computer vision research. In the current field of 3D computer vision, most datasets solely focus on a relatively limited set of tasks, which complicates the concurrent training requirements of various downstream tasks. This makes the training of multi-objective networks difficult to proceed with, which further hinders the development of foundation models in the 3D vision field. In this paper, we introduce VEnvision3D, a large 3D synthetic perception dataset for multi-task learning, including depth completion, segmentation, upsampling, place recognition, and 3D reconstruction. Since the data for each task was collected in the same scenarios, tasks are inherently aligned in terms of the utilized data. Therefore, such a unique attribute can assist in exploring the potential for the multi-task model and even the foundation model without separate training methods. Several new benchmarks based on the characteristics of the proposed dataset were presented. Extensive studies were performed on end-to-end models, revealing new observations, challenges, and opportunities for future research. In addition, we designed a straightfoward multi-task network to uncover the ability that VEnvision3D can offer for the foundation model. Our dataset and code will be open-sourced upon acceptance.

翻译：构建统一的通用多任务基础模型已成为计算机视觉研究的关键挑战。在当前的3D计算机视觉领域，大多数数据集仅集中于相对有限的任务集合，这使得各种下游任务的并行训练需求变得复杂化。多目标网络的训练难以推进，进一步阻碍了3D视觉领域基础模型的发展。本文提出VEnvision3D，一个面向多任务学习的大规模3D合成感知数据集，涵盖深度补全、分割、上采样、位置识别和三维重建等任务。由于各任务的数据均在相同场景中采集，任务在所用数据层面天然对齐。因此，这一独特属性可协助探索多任务模型乃至基础模型的潜力，无需采用独立训练方法。基于所提出数据集的特征，我们提出了若干新基准。通过对端到端模型进行广泛研究，揭示了新发现、挑战及未来研究机遇。此外，我们设计了一个简洁的多任务网络，以揭示VEnvision3D为基础模型提供的能力。我们的数据集和代码将在论文接收后开源。