The focal point of egocentric video understanding is modelling hand-object interactions. Standard models, e.g. CNNs or Vision Transformers, which receive RGB frames as input perform well. However, their performance improves further by employing additional input modalities that provide complementary cues, such as object detections, optical flow, audio, etc. The added complexity of the modality-specific modules, on the other hand, makes these models impractical for deployment. The goal of this work is to retain the performance of such a multimodal approach, while using only the RGB frames as input at inference time. We demonstrate that for egocentric action recognition on the Epic-Kitchens and the Something-Something datasets, students which are taught by multimodal teachers tend to be more accurate and better calibrated than architecturally equivalent models trained on ground truth labels in a unimodal or multimodal fashion. We further adopt a principled multimodal knowledge distillation framework, allowing us to deal with issues which occur when applying multimodal knowledge distillation in a naive manner. Lastly, we demonstrate the achieved reduction in computational complexity, and show that our approach maintains higher performance with the reduction of the number of input views. We release our code at https://github.com/gorjanradevski/multimodal-distillation.

翻译：自我中心视频理解的焦点在于建模手部与物体的交互。标准模型（如CNN或视觉Transformer）以RGB视频帧作为输入时表现良好。然而，通过引入提供互补线索的额外输入模态（如物体检测、光流、音频等），其性能可进一步提升。但另一方面，这些模态专用模块带来的附加复杂性使得模型难以实际部署。本工作的目标是在仅使用RGB帧作为推理输入的前提下，保持此类多模态方法的性能。我们证明，在Epic-Kitchens和Something-Something数据集上的自我中心动作识别任务中，由多模态教师指导的学生模型往往比在单模态或多模态方式下基于真实标签训练的结构等效模型具有更高的准确率和更好的校准性。我们进一步采用了一种原则性的多模态知识蒸馏框架，从而能够处理以朴素方式应用多模态知识蒸馏时出现的问题。最后，我们展示了计算复杂度的降低效果，并表明在减少输入视角数量的情况下，我们的方法仍能保持更高的性能。我们的代码发布于 https://github.com/gorjanradevski/multimodal-distillation。