This paper presents a Tri-branch Neural Fusion (TNF) approach designed for classifying multimodal medical images and tabular data. It also introduces two solutions to address the challenge of label inconsistency in multimodal classification. Traditional methods in multi-modality medical data classification often rely on single-label approaches, typically merging features from two distinct input modalities. This becomes problematic when features are mutually exclusive or labels differ across modalities, leading to reduced accuracy. To overcome this, our TNF approach implements a tri-branch framework that manages three separate outputs: one for image modality, another for tabular modality, and a third hybrid output that fuses both image and tabular data. The final decision is made through an ensemble method that integrates likelihoods from all three branches. We validate the effectiveness of TNF through extensive experiments, which illustrate its superiority over traditional fusion and ensemble methods in various convolutional neural networks and transformer-based architectures across multiple datasets.

翻译：本文提出了一种三分支神经融合（Tri-branch Neural Fusion，TNF）方法，用于对多模态医学图像与表格数据进行分类。同时，针对多模态分类中标签不一致的挑战，本文引入了两种解决方案。传统多模态医学数据分类方法通常采用单标签策略，一般从两种不同输入模态中融合特征。当特征互斥或不同模态标签不一致时，这种方法会导致分类精度下降。为克服上述局限性，TNF方法构建了一个三分支框架，管理三类独立输出：图像模态输出、表格模态输出，以及融合图像与表格数据的混合输出。最终决策通过集成方法实现，综合三个分支的似然值。通过大量实验验证了TNF方法的有效性，结果表明，在多种卷积神经网络与基于Transformer的架构上，TNF在多个数据集上均优于传统融合与集成方法。