We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). More than 800 participants from 13 cities worldwide performed these activities in 131 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,422 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources will be open sourced to fuel new research in the community.

翻译：我们提出了Ego-Exo4D，一个多样化、大规模的多模态多视角视频数据集及基准挑战。Ego-Exo4D聚焦于同时采集的第一人称和第三人称视角下的技能性人类活动（如体育、音乐、舞蹈、自行车维修）视频。来自全球13个城市的800余名参与者在131种不同的自然场景中完成这些活动，生成了时长从1至42分钟不等的长视频片段，视频总时长达到1,422小时。该数据集的多模态特性前所未有：视频伴随有多声道音频、眼动追踪、3D点云、相机位姿、惯性测量单元（IMU）以及多组配对语言描述——其中包括由教练和教师制作的、专用于技能活动领域的创新性"专家解说"。为推动技能性人类活动第一人称视频理解的边界，我们还提出了一套基准任务及其标注，涵盖细粒度活动理解、熟练度评估、跨视角转换以及3D手部/身体姿态。所有资源将开源，以促进社区的新研究。