In this work, we propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset. Based on the underlying principle of focal loss, which captures the relationship between tail (scarce) classes and their prediction difficulties, we propose an exponentially decaying variant of focal loss for our current task. It initially emphasizes learning from the hard misclassified examples and gradually adapts to the entire range of examples in the dataset. This annealing process encourages the model to strike a balance between focusing on the sparse set of hard samples, while still leveraging the information provided by the easier ones. Additionally, we opt for the late fusion strategy to combine the resultant probability distributions from RGB and Depth modalities for final action prediction. Experimental evaluations on the MECCANO dataset demonstrate the effectiveness of our approach.

翻译：本文提出了一种面向多模态动作识别的集成建模方法。我们利用一种针对MECCANO[21]数据集长尾分布定制的焦点损失变体，独立训练各模态模型。基于焦点损失的核心原理（该原理捕捉尾部（稀缺）类别与其预测难度之间的关系），我们针对当前任务提出了一种指数衰减的焦点损失变体。该变体首先强调从难以分类的错误样本中学习，然后逐步适应数据集中的全部样本范围。这种退火过程有助于模型在聚焦于稀疏的困难样本集与利用较易样本信息之间取得平衡。此外，我们采用后期融合策略，融合RGB模态和深度模态输出的概率分布以进行最终动作预测。在MECCANO数据集上的实验评估证明了我们方法的有效性。