The quadratic complexity of self-attention in Transformers has hindered the processing of long text. To alleviate this problem, previous works have proposed to sparsify the attention matrix, taking advantage of the observation that crucial information about a token can be derived from its neighbors. These methods typically combine one or another form of local attention and global attention. Such combinations introduce abrupt changes in contextual granularity when going from local to global, which may be undesirable. We believe that a smoother transition could potentially enhance model's ability to capture long-context dependencies. In this study, we introduce Fovea Transformer, a long-context focused transformer that addresses the challenges of capturing global dependencies while maintaining computational efficiency. To achieve this, we construct a multi-scale tree from the input sequence, and use representations of context tokens with a progressively coarser granularity in the tree, as their distance to the query token increases. We evaluate our model on three long-context summarization tasks\footnote{Our code is publicly available at: \textit{https://github.com/ZiweiHe/Fovea-Transformer}}. It achieves state-of-the-art performance on two of them, and competitive results on the third with mixed improvement and setback of the evaluation metrics.

翻译：Transformer中自注意力机制的二次复杂度阻碍了长文本的处理。为缓解该问题，前人工作提出利用相邻令牌可提供关键信息的观察结果来稀疏化注意力矩阵。这些方法通常结合某种形式的局部注意力和全局注意力。这种组合在从局部到全局的转换中会导致上下文粒度的突变，这可能是不可取的。我们认为更平滑的过渡有望增强模型捕获长上下文依赖的能力。本研究引入Fovea Transformer，一种专注于长上下文的Transformer，在保持计算效率的同时解决全局依赖捕获的挑战。为此，我们从输入序列构建多尺度树，并随着上下文令牌与查询令牌距离的增加，在树中使用粒度逐步变粗的上下文令牌表示。我们在三个长上下文摘要任务上评估模型\footnote{代码公开于：\textit{https://github.com/ZiweiHe/Fovea-Transformer}}。模型在其中两个任务上达到最先进水平，在第三个任务上取得竞争性结果，但评估指标呈现混合改进与回退。