Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.

翻译：多样性在数据选择中扮演关键角色，通过减少冗余与重复可在固定数据预算下提升模型性能。然而，优化多样性本身极具挑战性，因其作为集合级属性依赖于数据点间的交互而非单个样本特征。现有方法通常依赖代理指标或近似策略，但往往难以保证足够多样化的子集。本文基于G-Vendi分数提出概率多样化框架，通过指数梯度下降直接优化多样性。本方法生成的子集多样性显著优于随机采样，在50万样本子集上实现G-Vendi分数提升+489。我们在FineWeb和DCLM数据集上评估该方法，其表现持续优于现有方案。值得注意的是，SPOKES（仅多样性策略）在DCLM和FineWeb上分别将下游任务平均性能较随机采样提升+0.4和+0.5个百分点。更重要的是，质量与多样性的联合优化可实现最优性能：SPOKES在DCLM和FineWeb上分别取得+1.5和+1.4个百分点的增益，超越包括语义去重和质量过滤在内的所有基线方法。