Visual Question Answering on 3D Point Cloud (VQA-3D) is an emerging yet challenging field that aims at answering various types of textual questions given an entire point cloud scene. To tackle this problem, we propose the CLEVR3D, a large-scale VQA-3D dataset consisting of 171K questions from 8,771 3D scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Through such a manner, we initially generated 44K questions from 1,333 real-world scenes. Moreover, a more challenging setup is proposed to remove the confounding bias and adjust the context from a common-sense layout. Such a setup requires the network to achieve comprehensive visual understanding when the 3D scene is different from the general co-occurrence context (e.g., chairs always exist with tables). To this end, we further introduce the compositional scene manipulation strategy and generate 127K questions from 7,438 augmented 3D scenes, which can improve VQA-3D models for real-world comprehension. Built upon the proposed dataset, we baseline several VQA-3D models, where experimental results verify that the CLEVR3D can significantly boost other 3D scene understanding tasks. Our code and dataset will be made publicly available at https://github.com/yanx27/CLEVR3D.

翻译：三维点云视觉问答（VQA-3D）是一个新兴且具有挑战性的领域，旨在针对整个点云场景回答各类文本问题。为解决该问题，本文提出CLEVR3D，一个包含8,771个三维场景中171K个问题的大规模VQA-3D数据集。具体而言，我们开发了一种利用三维场景图结构生成多样化推理问题的问题引擎，涵盖物体属性（即尺寸、颜色和材质）及其空间关系类问题。通过这种方式，我们初步从1,333个真实场景生成了44K个问题。此外，我们提出了更具挑战性的设置，以消除混杂偏差并将上下文调整至常识性布局之外。该设置要求网络在三维场景偏离常规共现语境（例如椅子常与桌子共存）时实现全面的视觉理解。为此，我们进一步引入组合场景操作策略，从7,438个增强三维场景生成127K个问题，从而提升VQA-3D模型对真实世界的理解能力。基于所提数据集，我们对多个VQA-3D模型进行了基准测试，实验结果表明CLEVR3D能显著提升其他三维场景理解任务。我们的代码和数据集将在https://github.com/yanx27/CLEVR3D公开发布。