A social highlighter's most useful signal -- which passages a crowd of readers marks -- exists only for documents people have already read. Can the aggregate crowd salience of a document be predicted from its text before its marks accumulate? Prior work on this data found that zero-shot language models recover highlight locations worse than a trivial lead (position) baseline, so we ask whether a model trained on the highlight corpus can beat that baseline. Using a pre-registered ladder of models and a by-document cluster bootstrap, we find a small but robust edge: a logistic ranker over sentence embeddings and positional/contextual features beats the lead baseline by +0.044 average precision (95% CI [+0.029, +0.058]; clears a pre-registered margin delta=0.03 in 97% of resamples, and stable across pipeline re-runs). Two unsupervised extractive baselines (centroid, LexRank-style centrality) lose to lead, and the trained model beats them by +0.108, so the edge is not recovered by generic unsupervised proxies -- it reflects learning from real reader marks. In product terms, precision@3 rises from 0.25 to 0.39 (+55% relative) and the model beats lead on 69% of documents. An ablation attributes the edge to the raw embedding (+0.014) and training augmentation (+0.010), each with a positive CI. The edge is not a temporal-generalization failure, and we find no evidence that content drift or near-duplicate leakage explains it. A standardized regression shows the advantage is governed mainly by document popularity (lower popularity, larger edge) and by label reliability. It nearly vanishes only on the most popular content; there it is the lead baseline that strengthens, not the model that weakens. Because our evaluation conditions on documents that eventually accumulated readers, these results are a retrospective cold-start simulation.
翻译:摘要:社交高亮工具最关键的信号——读者群体标记的段落——仅存在于已被阅读的文档中。能否在标记累积之前,从文档文本预测其整体众包显著性?先前针对该数据的研究发现,零样本语言模型对高亮位置的还原效果甚至不如简单的前置(位置)基线。本研究旨在探究:基于高亮语料库训练的模型能否超越该基线?通过预先注册的模型阶梯和按文档分组的聚类自助法,我们发现了一个虽小但稳健的优势:基于句子嵌入与位置/上下文特征的逻辑排序器在前置基线上实现了平均精确率提升+0.044(95%置信区间[+0.029,+0.058];在97%的重采样中超过预先注册的边界δ=0.03,且跨流水线复现保持稳定)。两种无监督提取基线(质心法、LexRank式中心性)均逊于前置基线,而训练模型超出其+0.108,表明该优势并非源于通用无监督代理——而是源自对真实读者标记的学习。从产品角度,精确率@3从0.25提升至0.39(相对+55%),模型在69%的文档上超越前置基线。消融实验表明,该优势分别来自原始嵌入(+0.014)和训练增强(+0.010),两者置信区间均为正。该优势并非时间泛化失败所致,且无证据表明内容漂移或近似重复泄漏可解释该现象。标准化回归显示,优势主要受文档流行度(流行度越低,优势越大)和标签可靠性调控。该优势仅在最高流行度内容上近乎消失——此时是前置基线增强,而非模型性能下降。由于评估条件限定于最终累积到读者的文档,这些结果构成了一种回顾性冷启动模拟。