In this paper, we propose the Hierarchical Document Transformer (HDT), a novel sparse Transformer architecture tailored for structured hierarchical documents. Such documents are extremely important in numerous domains, including science, law or medicine. However, most existing solutions are inefficient and fail to make use of the structure inherent to documents. HDT exploits document structure by introducing auxiliary anchor tokens and redesigning the attention mechanism into a sparse multi-level hierarchy. This approach facilitates information exchange between tokens at different levels while maintaining sparsity, thereby enhancing computational and memory efficiency while exploiting the document structure as an inductive bias. We address the technical challenge of implementing HDT's sample-dependent hierarchical attention pattern by developing a novel sparse attention kernel that considers the hierarchical structure of documents. As demonstrated by our experiments, utilizing structural information present in documents leads to faster convergence, higher sample efficiency and better performance on downstream tasks.

翻译：本文提出了一种新颖的稀疏Transformer架构——层次化文档Transformer（HDT），专为具有结构化层次的文档而设计。此类文档在科学、法律和医学等诸多领域都极为重要。然而，现有的大多数解决方案效率低下，未能充分利用文档固有的结构信息。HDT通过引入辅助锚点标记，并将注意力机制重新设计为稀疏的多层次结构，从而有效利用文档结构。这种方法促进了不同层次标记之间的信息交换，同时保持了稀疏性，从而在利用文档结构作为归纳偏置的同时，提高了计算和内存效率。我们通过开发一种新颖的稀疏注意力核来解决HDT中样本依赖的层次化注意力模式的技术挑战，该核考虑了文档的层次结构。我们的实验表明，利用文档中存在的结构信息能够带来更快的收敛速度、更高的样本效率以及在下游任务上更好的性能。