Millimeter Wave (mmWave) Radar is gaining popularity as it can work in adverse environments like smoke, rain, snow, poor lighting, etc. Prior work has explored the possibility of reconstructing 3D skeletons or meshes from the noisy and sparse mmWave Radar signals. However, it is unclear how accurately we can reconstruct the 3D body from the mmWave signals across scenes and how it performs compared with cameras, which are important aspects needed to be considered when either using mmWave radars alone or combining them with cameras. To answer these questions, an automatic 3D body annotation system is first designed and built up with multiple sensors to collect a large-scale dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images in different scenes and skeleton/mesh annotations for humans in the scenes. With this dataset, we train state-of-the-art methods with inputs from different sensors and test them in various scenarios. The results demonstrate that 1) despite the noise and sparsity of the generated point clouds, the mmWave radar can achieve better reconstruction accuracy than the RGB camera but worse than the depth camera; 2) the reconstruction from the mmWave radar is affected by adverse weather conditions moderately while the RGB(D) camera is severely affected. Further, analysis of the dataset and the results shadow insights on improving the reconstruction from the mmWave radar and the combination of signals from different sensors.

翻译：毫米波（mmWave）雷达因可在恶劣环境下工作，如烟雾、雨雪、光线不足等而逐渐受到人们的关注。之前的研究探讨了利用噪声和稀疏的mmWave雷达信号重建3D骨架或网格的可能性。然而，目前尚不清楚我们能在不同场景下从mmWave信号中精确地重建3D人体，以及在这方面和摄像机相比性能如何。而这些对于使用mmWave雷达单独或将其与摄像机相结合都是重要的方面。为了回答这些问题，首先设计并建立了自动3D人体注释系统，并使用多个传感器收集同步和校准的mmWave雷达点云和RGB（D）图像以形成大型数据集。该数据集包含不同场景中人体的骨架/网格注释。利用该数据集，我们使用不同传感器输入训练了最先进的方法，并在各种场景中进行测试。结果表明：1）尽管所生成的点云噪声和稀疏，但mmWave雷达可以比RGB摄像机实现更好的重建精度，但比深度摄像机差；2）mmWave雷达的重建受恶劣天气条件的影响适中，而RGB（D）摄像机则受到严重影响。此外，对数据集和结果的分析揭示了改善从mmWave雷达和不同传感器信号组合中的重建的洞见。