In this work, we propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset. Based on the underlying principle of focal loss, which captures the relationship between tail (scarce) classes and their prediction difficulties, we propose an exponentially decaying variant of focal loss for our current task. It initially emphasizes learning from the hard misclassified examples and gradually adapts to the entire range of examples in the dataset. This annealing process encourages the model to strike a balance between focusing on the sparse set of hard samples, while still leveraging the information provided by the easier ones. Additionally, we opt for the late fusion strategy to combine the resultant probability distributions from RGB and Depth modalities for final action prediction. Experimental evaluations on the MECCANO dataset demonstrate the effectiveness of our approach.

翻译：本文提出了一种用于多模态动作识别的集成建模方法。我们独立训练各模态模型，采用针对MECCANO[21]数据集长尾分布问题定制的focal loss变体。基于focal loss捕捉尾类（稀缺类）与预测难度之间关系的基本原理，我们针对当前任务提出了一种指数衰减型focal loss变体。该损失函数初期强调对难分类误例的学习，并逐渐适应数据集中所有类型的样本。这种退火过程促使模型在关注稀疏的困难样本与利用简单样本信息之间取得平衡。此外，我们采用晚期融合策略，融合RGB和深度模态的概率分布以进行最终动作预测。在MECCANO数据集上的实验评估验证了我们方法的有效性。