We study object interaction anticipation in egocentric videos. This task requires an understanding of the spatiotemporal context formed by past actions on objects, coined action context. We propose TransFusion, a multimodal transformer-based architecture. It exploits the representational power of language by summarising the action context. TransFusion leverages pre-trained image captioning and vision-language models to extract the action context from past video frames. This action context together with the next video frame is processed by the multimodal fusion module to forecast the next object interaction. Our model enables more efficient end-to-end learning. The large pre-trained language models add common sense and a generalisation capability. Experiments on Ego4D and EPIC-KITCHENS-100 show the effectiveness of our multimodal fusion model. They also highlight the benefits of using language-based context summaries in a task where vision seems to suffice. Our method outperforms state-of-the-art approaches by 40.4% in relative terms in overall mAP on the Ego4D test set. We validate the effectiveness of TransFusion via experiments on EPIC-KITCHENS-100. Video and code are available at: https://eth-ait.github.io/transfusion-proj/.

翻译：我们研究以自我为中心视频中的物体交互预测任务。该任务需要理解由对物体的历史动作形成的时空上下文，我们称之为动作上下文。我们提出TransFusion，一种基于Transformer的多模态架构。它通过总结动作上下文来利用语言的强大表征能力。TransFusion利用预训练的图像描述生成模型和视觉-语言模型从历史视频帧中提取动作上下文。该动作上下文与下一视频帧一起由多模态融合模块处理，以预测下一个物体交互。我们的模型支持更高效的端到端学习。大规模预训练语言模型引入了常识和泛化能力。在Ego4D和EPIC-KITCHENS-100上的实验展示了我们多模态融合模型的有效性。这些实验还突显了在看似仅凭视觉即可完成的任务中使用基于语言的上下文总结的优势。我们的方法在Ego4D测试集上的整体mAP相对优于现有最佳方法40.4%。我们通过EPIC-KITCHENS-100上的实验验证了TransFusion的有效性。视频和代码见：https://eth-ait.github.io/transfusion-proj/。