Transformers achieve promising performance in document understanding because of their high effectiveness and still suffer from quadratic computational complexity dependency on the sequence length. General efficient transformers are challenging to be directly adapted to model document. They are unable to handle the layout representation in documents, e.g. word, line and paragraph, on different granularity levels and seem hard to achieve a good trade-off between efficiency and performance. To tackle the concerns, we propose Fast-StrucTexT, an efficient multi-modal framework based on the StrucTexT algorithm with an hourglass transformer architecture, for visual document understanding. Specifically, we design a modality-guided dynamic token merging block to make the model learn multi-granularity representation and prunes redundant tokens. Additionally, we present a multi-modal interaction module called Symmetry Cross Attention (SCA) to consider multi-modal fusion and efficiently guide the token mergence. The SCA allows one modality input as query to calculate cross attention with another modality in a dual phase. Extensive experiments on FUNSD, SROIE, and CORD datasets demonstrate that our model achieves the state-of-the-art performance and almost 1.9X faster inference time than the state-of-the-art methods.

翻译：Transformer凭借其高效性在文档理解中取得了优异性能，但仍受限于序列长度二次复杂度的计算瓶颈。通用高效Transformer难以直接适用于文档建模，无法处理文档中不同粒度级别（如词、行、段落）的布局表示，且难以在效率与性能间实现良好平衡。为解决上述问题，本文提出Fast-StrucTexT——一种基于StrucTexT算法、采用沙漏式Transformer架构的高效多模态视觉文档理解框架。具体而言，我们设计了模态引导的动态令牌合并模块，使模型能够学习多粒度表示并剪枝冗余令牌；此外，提出了名为对称交叉注意力（SCA）的多模态交互模块，在实现多模态融合的同时有效指导令牌合并。SCA允许以单模态输入作为查询，在双阶段中与另一模态计算交叉注意力。在FUNSD、SROIE和CORD数据集上的大量实验表明，本模型在实现最优性能的同时，推理速度相比现有最优方法提升约1.9倍。