Multimodal learning from document data has achieved great success lately as it allows to pre-train semantically meaningful features as a prior into a learnable downstream task. In this paper, we approach the document classification problem by learning cross-modal representations through language and vision cues, considering intra- and inter-modality relationships. Instead of merging features from different modalities into a joint representation space, the proposed method exploits high-level interactions and learns relevant semantic information from effective attention flows within and across modalities. The proposed learning objective is devised between intra- and inter-modality alignment tasks, where the similarity distribution per task is computed by contracting positive sample pairs while simultaneously contrasting negative ones in the joint representation space}. Extensive experiments on public document classification datasets demonstrate the effectiveness and the generality of our model on low-scale and large-scale datasets.

翻译：多模态文档数据学习近期取得了显著成功，因其能够将语义有意义的特征作为先验知识预训练到可学习的下游任务中。本文通过语言与视觉线索学习跨模态表征来解决文档分类问题，同时考虑模态内与模态间关系。所提方法并非将不同模态的特征融合到联合表示空间，而是通过跨模态及模态内的高效注意力流，利用高层交互学习相关语义信息。设计的训练目标在模态内与模态间对齐任务之间建立，通过收缩正样本对并同时对比负样本对，在联合表示空间中计算每个任务的相似度分布。在公开文档分类数据集上的大量实验证明，本模型在低规模和大规模数据集上均具有有效性和泛化性。