4D human perception plays an essential role in a myriad of applications, such as home automation and metaverse avatar simulation. However, existing solutions which mainly rely on cameras and wearable devices are either privacy intrusive or inconvenient to use. To address these issues, wireless sensing has emerged as a promising alternative, leveraging LiDAR, mmWave radar, and WiFi signals for device-free human sensing. In this paper, we propose MM-Fi, the first multi-modal non-intrusive 4D human dataset with 27 daily or rehabilitation action categories, to bridge the gap between wireless sensing and high-level human perception tasks. MM-Fi consists of over 320k synchronized frames of five modalities from 40 human subjects. Various annotations are provided to support potential sensing tasks, e.g., human pose estimation and action recognition. Extensive experiments have been conducted to compare the sensing capacity of each or several modalities in terms of multiple tasks. We envision that MM-Fi can contribute to wireless sensing research with respect to action recognition, human pose estimation, multi-modal learning, cross-modal supervision, and interdisciplinary healthcare research.

翻译：4D人体感知在众多应用中扮演着关键角色，例如家庭自动化与元宇宙虚拟人仿真。然而，现有主要依赖摄像头和可穿戴设备的解决方案要么存在隐私侵犯问题，要么使用不便。为解决这些挑战，无线感知技术作为一种极具前景的替代方案应运而生，通过利用激光雷达、毫米波雷达和WiFi信号实现无设备人体感知。本文提出MM-Fi——首个包含27种日常或康复动作类别的多模态非侵入式4D人体数据集，旨在弥合无线感知与高级人体感知任务之间的鸿沟。MM-Fi包含来自40名受试者的超过32万帧同步数据，涵盖五种模态，并提供多种标注以支持潜在感知任务（如人体姿态估计和动作识别）。我们开展了大量实验，从多任务维度评估了每种或多种模态的感知能力。我们预期MM-Fi将在动作识别、人体姿态估计、多模态学习、跨模态监督以及跨学科健康医疗研究等领域推动无线感知研究的发展。