Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token. However, not all tokens are equally important, especially for longer documents. We propose CoLT5, a long-input Transformer model that builds on this intuition by employing conditional computation, devoting more resources to important tokens in both feedforward and attention layers. We show that CoLT5 achieves stronger performance than LongT5 with much faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. Moreover, CoLT5 can effectively and tractably make use of extremely long inputs, showing strong gains up to 64k input length.

翻译：许多自然语言处理任务受益于长输入，但使用Transformer处理长文档成本高昂——这不仅源于二次注意力复杂度，还因为需要对每个token应用前馈和投影层。然而，并非所有token都具有同等重要性，尤其对于长文档而言。我们提出CoLT5，一种基于这一直觉的长输入Transformer模型，通过采用条件计算，在前馈层和注意力层中为重要token分配更多计算资源。实验表明，CoLT5在实现更快训练和推理的同时，性能显著优于LongT5，并在长序列SCROLLS基准测试中达到最佳水平。此外，CoLT5能够高效且可控地利用极长输入，在64k输入长度下仍展现出显著性能提升。