The field of 4D point cloud understanding is rapidly developing with the goal of analyzing dynamic 3D point cloud sequences. However, it remains a challenging task due to the sparsity and lack of texture in point clouds. Moreover, the irregularity of point cloud poses a difficulty in aligning temporal information within video sequences. To address these issues, we propose a novel cross-modal knowledge transfer framework, called X4D-SceneFormer. This framework enhances 4D-Scene understanding by transferring texture priors from RGB sequences using a Transformer architecture with temporal relationship mining. Specifically, the framework is designed with a dual-branch architecture, consisting of an 4D point cloud transformer and a Gradient-aware Image Transformer (GIT). During training, we employ multiple knowledge transfer techniques, including temporal consistency losses and masked self-attention, to strengthen the knowledge transfer between modalities. This leads to enhanced performance during inference using single-modal 4D point cloud inputs. Extensive experiments demonstrate the superior performance of our framework on various 4D point cloud video understanding tasks, including action recognition, action segmentation and semantic segmentation. The results achieve 1st places, i.e., 85.3% (+7.9%) accuracy and 47.3% (+5.0%) mIoU for 4D action segmentation and semantic segmentation, on the HOI4D challenge\footnote{\url{http://www.hoi4d.top/}.}, outperforming previous state-of-the-art by a large margin. We release the code at https://github.com/jinglinglingling/X4D

翻译：四维点云理解领域的目标是分析动态三维点云序列，近年来发展迅速。然而，由于点云的稀疏性和缺乏纹理信息，该任务仍具有挑战性。此外，点云的不规则性使得视频序列中的时间信息对齐变得困难。为解决这些问题，我们提出了一种新颖的跨模态知识迁移框架，命名为X4D-SceneFormer。该框架通过利用基于时间关系挖掘的Transformer架构，将RGB序列中的纹理先验知识迁移至四维场景理解中。具体而言，该框架采用双分支架构设计，包括四维点云Transformer和梯度感知图像Transformer（GIT）。在训练过程中，我们采用多种知识迁移技术，包括时间一致性损失和掩码自注意力机制，以增强模态间的知识迁移，从而在使用单模态四维点云输入进行推理时提升性能。大量实验表明，我们的框架在多种四维点云视频理解任务（包括动作识别、动作分割和语义分割）中均展现出卓越性能。在HOI4D挑战赛（网址：http://www.hoi4d.top/）中，结果取得第一名：四维动作分割准确率达85.3%（提升了7.9%），语义分割平均交并比（mIoU）达47.3%（提升了5.0%），大幅超越先前最先进方法。我们已在https://github.com/jinglinglingling/X4D 发布代码。