A core limitation of standard softmax attention is that it does not define a notion of absolute query--key relevance: attention weights are obtained by redistributing a fixed unit mass across all keys according to their relative scores. As a result, relevance is defined only relative to competing keys, and irrelevant keys cannot be explicitly rejected. We introduce Multiscreen, a language-model architecture built around a mechanism we call screening, which enables absolute query--key relevance. Instead of redistributing attention across all keys, screening evaluates each key against an explicit threshold, discarding irrelevant keys and aggregating the remaining keys, thereby removing global competition among keys. Across experiments, Multiscreen achieves comparable validation loss with approximately 40% fewer parameters than a Transformer baseline and enables stable optimization at substantially larger learning rates. It maintains strong performance in long-context perplexity and shows little to no degradation in retrieval performance well beyond the training context length. Notably, even at the training context length, a Multiscreen model with approximately 92% fewer parameters consistently outperforms a larger Transformer in retrieval accuracy. Finally, Multiscreen reduces inference latency by up to 3.2$\times$ at 100K context length.

翻译：标准softmax注意力的一个核心局限在于它无法定义绝对的查询-键相关性：注意力权重是通过将固定单位质量根据相对得分在全体键上重新分配而得到的。因此，相关性仅相对于竞争键进行定义，无关键无法被明确排除。我们提出Multiscreen——一种围绕名为"筛分"的机制构建的语言模型架构，该机制实现了绝对的查询-键相关性。筛分并非在全体键上重新分配注意力，而是针对每个键与显式阈值进行比较，舍弃无关键并聚合剩余键，从而消除了键之间的全局竞争。在多项实验中，相比Transformer基线，Multiscreen在参数减少约40%的情况下实现了可比的验证损失，并能在显著更大的学习率下保持稳定优化。该模型在长上下文困惑度上保持强劲性能，且在远超训练上下文长度的检索任务中几乎不出现性能退化。值得注意的是，即使在训练上下文长度下，参数减少约92%的Multiscreen模型在检索准确率上始终优于更大的Transformer模型。最后，在100K上下文长度下，Multiscreen将推理延迟降低了最高3.2倍。