Millimeter Wave (mmWave) Radar is gaining popularity as it can work in adverse environments like smoke, rain, snow, poor lighting, etc. Prior work has explored the possibility of reconstructing 3D skeletons or meshes from the noisy and sparse mmWave Radar signals. However, it is unclear how accurately we can reconstruct the 3D body from the mmWave signals across scenes and how it performs compared with cameras, which are important aspects needed to be considered when either using mmWave radars alone or combining them with cameras. To answer these questions, an automatic 3D body annotation system is first designed and built up with multiple sensors to collect a large-scale dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images in different scenes and skeleton/mesh annotations for humans in the scenes. With this dataset, we train state-of-the-art methods with inputs from different sensors and test them in various scenarios. The results demonstrate that 1) despite the noise and sparsity of the generated point clouds, the mmWave radar can achieve better reconstruction accuracy than the RGB camera but worse than the depth camera; 2) the reconstruction from the mmWave radar is affected by adverse weather conditions moderately while the RGB(D) camera is severely affected. Further, analysis of the dataset and the results shadow insights on improving the reconstruction from the mmWave radar and the combination of signals from different sensors.

翻译：毫米波雷达因其能在烟雾、雨雪、弱光等恶劣环境中工作而日益受到关注。已有研究探索了从噪声大且稀疏的毫米波雷达信号中重建三维骨骼或网格的可能性。然而，目前尚不清楚在不同场景下，毫米波信号对人体三维重建的准确程度如何，以及与相机相比的性能表现——这些是在单独使用毫米波雷达或将其与相机结合时必须考虑的重要因素。为解答这些问题，我们首先设计并构建了一套自动三维人体标注系统，该系统配备多个传感器以采集大规模数据集。该数据集包含不同场景下经过同步与标定的毫米波雷达点云、RGB(D)图像，以及场景中人体的骨骼/网格标注。基于此数据集，我们训练了基于不同传感器输入的当前最优方法，并在多种场景下进行测试。结果表明：1）尽管生成的点云存在噪声和稀疏性，毫米波雷达的重建精度仍优于RGB相机，但逊于深度相机；2）毫米波雷达的重建受恶劣天气条件影响较小，而RGB(D)相机则受严重影响。此外，对数据集及结果的分析为改进毫米波雷达重建技术及多传感器信号融合提供了重要启示。