Multi-vector late-interaction retrievers such as ColBERT achieve state-of-the-art retrieval quality, but their query-time cost is dominated by exhaustively computing token-level MaxSim interactions for every candidate document. While approximating late interaction with single-vector representations reduces cost, it often incurs substantial accuracy loss. We introduce Col-Bandit, a query-time pruning algorithm that reduces this computational burden by casting reranking as a finite-population Top-$K$ identification problem. Col-Bandit maintains uncertainty-aware bounds over partially observed document scores and adaptively reveals only the (document, query token) MaxSim entries needed to determine the top results under statistical decision bounds with a tunable relaxation. Unlike coarse-grained approaches that prune entire documents or tokens offline, Col-Bandit sparsifies the interaction matrix on the fly. It operates as a zero-shot, drop-in layer over standard multi-vector systems, requiring no index modifications, offline preprocessing, or model retraining. Experiments on textual (BEIR) and multimodal (REAL-MM-RAG) benchmarks show that Col-Bandit preserves ranking fidelity while reducing MaxSim FLOPs by up to 5$\times$, indicating that dense late-interaction scoring contains substantial redundancy that can be identified and pruned efficiently at query time.

翻译：多向量延迟交互检索器（如ColBERT）能够实现最先进的检索质量，但其查询时成本主要源于为每个候选文档详尽计算词元级MaxSim交互。虽然使用单向量表示来近似延迟交互可以降低成本，但这通常会带来显著的精度损失。我们提出了Col-Bandit，一种查询时剪枝算法，通过将重排序建模为一个有限总体Top-$K$识别问题来减轻这一计算负担。Col-Bandit基于部分观测到的文档分数维护不确定性感知的边界，并在可调松弛度的统计决策边界下，自适应地仅揭示确定顶部结果所需的（文档，查询词元）MaxSim条目。与离线剪枝整个文档或词元的粗粒度方法不同，Col-Bandit在运行时对交互矩阵进行稀疏化。它作为一个零样本、即插即用的层运行在标准多向量系统之上，无需修改索引、离线预处理或模型重新训练。在文本（BEIR）和多模态（REAL-MM-RAG）基准测试上的实验表明，Col-Bandit在保持排序保真度的同时，将MaxSim的浮点运算次数减少了高达5$\times$，这表明密集的延迟交互评分包含大量冗余，可以在查询时被高效识别并剪枝。