While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechanism. To address this, we propose NextLevelBERT, a Masked Language Model operating not on tokens, but on higher-level semantic representations in the form of text embeddings. We pretrain NextLevelBERT to predict the vector representation of entire masked text chunks and evaluate the effectiveness of the resulting document vectors on three task types: 1) Semantic Textual Similarity via zero-shot document embeddings, 2) Long document classification, 3) Multiple-choice question answering. We find that next level Masked Language Modeling is an effective technique to tackle long-document use cases and can outperform much larger embedding models as long as the required level of detail is not too high. We make model and code available.

翻译：尽管（大型）语言模型在过去几年中取得了显著进步，但由于底层注意力机制的二次方复杂度，它们在处理书籍等长序列时仍存在困难。为此，我们提出NextLevelBERT——一种不基于词元（token）而基于文本嵌入形式的高层级语义表征进行操作的掩码语言模型。我们预训练NextLevelBERT模型来预测完整掩码文本块的向量表征，并通过三类任务评估所得文档向量的有效性：1）基于零样本文档嵌入的语义文本相似度；2）长文档分类；3）多项选择问答。研究发现，这种层级掩码语言建模技术是解决长文档应用场景的有效手段，且在对细节粒度要求不高的条件下，其性能可超越体量更大的嵌入模型。我们已公开该模型及源代码。