A core limitation of standard softmax attention is that it does not define a notion of absolute query--key relevance: attention weights are obtained by redistributing a fixed unit mass across all keys according to their relative scores. As a result, relevance is defined only relative to competing keys, and irrelevant keys cannot be explicitly rejected. We introduce Multiscreen, a language-model architecture built around a mechanism we call screening, which enables absolute query--key relevance. Instead of redistributing attention across all keys, screening evaluates each key against an explicit threshold, discarding irrelevant keys and aggregating the remaining keys, thereby removing global competition among keys. Across experiments, Multiscreen achieves comparable validation loss with approximately 40% fewer parameters than a Transformer baseline, enables stable optimization at substantially larger learning rates, maintains strong performance in long-context perplexity, shows little to no degradation in retrieval performance even far beyond the training context length, and reduces inference latency by up to 3.2$\times$ at 100K context length.

翻译：标准softmax注意力的核心局限在于未定义绝对的查询-键相关性：注意力权重是通过根据相对分数将固定单位质量重新分配到所有键上获得的。因此，相关性仅相对于竞争键定义，无法显式拒绝无关键。我们提出Multiscreen——一种围绕我们称之为筛选的机制构建的语言模型架构，该机制实现了绝对的查询-键相关性。筛选不再将注意力重新分配到所有键上，而是根据显式阈值评估每个键，丢弃无关键并聚合剩余键，从而消除键之间的全局竞争。实验表明，与Transformer基线相比，Multiscreen在参数约减少40%的情况下实现可比的验证损失，在显著更大的学习率下实现稳定优化，在长上下文困惑度上保持强劲性能，即使在远超训练上下文长度时检索性能也几乎没有下降，并在100K上下文长度下将推理延迟降低至多3.2倍。