Though research has shown the complementarity of camera- and inertial-based data, datasets which offer both modalities remain scarce. In this paper, we introduce WEAR, an outdoor sports dataset for both vision- and inertial-based human activity recognition (HAR). The dataset comprises data from 18 participants performing a total of 18 different workout activities with untrimmed inertial (acceleration) and camera (egocentric video) data recorded at 10 different outside locations. Unlike previous egocentric datasets, WEAR provides a challenging prediction scenario marked by purposely introduced activity variations as well as an overall small information overlap across modalities. Provided benchmark results reveal that single-modality architectures each have different strengths and weaknesses in their prediction performance. Further, in light of the recent success of transformer-based temporal action localization models, we demonstrate their versatility by applying them in a plain fashion using vision, inertial and combined (vision + inertial) features as input. Results demonstrate both the applicability of vision-based transformers for inertial data and fusing both modalities by means of simple concatenation, with the combined approach (vision + inertial features) being able to produce the highest mean average precision and close-to-best F1-score. The dataset and code to reproduce experiments is publicly available via: https://mariusbock.github.io/wear/

翻译：摘要：尽管已有研究证明了摄像头与惯性数据的互补性，但同时提供两种模态的数据集仍然稀缺。本文提出了WEAR，一个面向户外运动的视觉与惯性人体活动识别（HAR）数据集。该数据集包含18名参与者在10个不同户外地点执行18种不同锻炼活动时采集的未经裁剪的惯性（加速度）与摄像头（第一人称视角视频）数据。与先前第一人称视角数据集不同，WEAR通过刻意引入活动变异及跨模态信息重叠度极低的设计，构建了具有挑战性的预测场景。提供的基准测试结果表明，单模态架构在预测性能上各有优劣。此外，鉴于基于Transformer的时序动作定位模型近期取得的成功，我们通过以视觉、惯性及融合（视觉+惯性）特征作为输入，采用朴素方式应用这些模型，验证了其多模态适用性。实验结果显示，基于视觉的Transformer对惯性数据具有可迁移性，且通过简单拼接即可实现双模态融合——融合方法（视觉+惯性特征）能获得最高平均精度均值及接近最优的F1分数。数据集及实验复现代码已公开于：https://mariusbock.github.io/wear/