Why does the low dimensionality of representations, typically $d\approx 1000$, not prevent modern embedding-based retrieval models from scaling to billions, or even trillions, of data points? To answer this question, we study maximal-margin embeddings in the following retrieval model, classically studied in communication complexity [PS86] and more recently in embedding-based retrieval [WBNL26]. Let $A\in \{0,1\}^{N\times n}$ be a matrix indicating whether each of $N$ queries is relevant to each of $n$ documents. We are interested in the largest margin $m>0,$ denoted by $\mathsf{m}^{\mathsf{rd}}(d, A),$ for which there exist unit norm embeddings of the queries and documents $\{U_j\}_{j = 1}^N, \{V_i\}_{i = 1}^n$ with the following property. $\langle U_j, V_i\rangle \ge m$ whenever $A_{ji} = 1$ and $\langle U_j, V_i\rangle \le -m$ otherwise. A large margin is a key proxy for representation quality: it controls both robustness to perturbations and compositional generalization across queries. Our main theorem establishes that the best possible margin without a restriction on the dimension, $\mathsf{m}^{\mathsf{rd}}(+\infty, A),$ can be nearly achieved in dimension $d = O(\mathsf{m}^{\mathsf{rd}}(+\infty, A)^{-2}\log n)$ which improves a theorem of [BDES02]. Together with a matching lower bound in Theorem 1.5, we conclude that when $A\in \{0,1\}^{\binom{n}{k}\times n}$ is the matrix containing all possible $k$-sparse rows once, dimension $d = O(k\log (n/k))$ is necessary and sufficient for the maximal possible margin $\mathsf{m}^{\mathsf{rd}}(+\infty, A) = Θ(k^{-1/2})$ in this setting. This fully resolves the setup of [WBNL26]. We also give several constructions for large margins when $d = o(k\log (n/k)).$ Finally, we empirically test the InfoNCE and sigmoid losses for producing large margin embeddings and demonstrate a clear advantage of the sigmoid loss.

翻译：为什么低维表示（通常$d\approx 1000$）不会阻碍现代基于嵌入的检索模型扩展到数十亿甚至万亿个数据点？为回答此问题，我们在以下检索模型中研究最大间隔嵌入——该模型在通信复杂度领域经典研究[PS86]中已提出，近期则被用于基于嵌入的检索[WBNL26]。设$A\in \{0,1\}^{N\times n}$为指示矩阵，表示$N$个查询与$n$个文档的相关性。我们关注最大间隔$m>0$（记作$\mathsf{m}^{\mathsf{rd}}(d, A)$），使得存在查询和文档的单位范数嵌入$\{U_j\}_{j = 1}^N, \{V_i\}_{i = 1}^n$满足：当$A_{ji} = 1$时$\langle U_j, V_i\rangle \ge m$，否则$\langle U_j, V_i\rangle \le -m$。大间隔是表示质量的关键代理指标：它同时控制对扰动的鲁棒性和跨查询的组合泛化能力。我们的主定理证明，无维度限制时可能的最优间隔$\mathsf{m}^{\mathsf{rd}}(+\infty, A)$可在维度$d = O(\mathsf{m}^{\mathsf{rd}}(+\infty, A)^{-2}\log n)$下近似实现，这改进了[BDES02]的定理。结合定理1.5中的匹配下界，我们得出结论：当$A\in \{0,1\}^{\binom{n}{k}\times n}$为包含所有可能$k$-稀疏行的矩阵时，维度$d = O(k\log (n/k))$是实现该设定下最大可能间隔$\mathsf{m}^{\mathsf{rd}}(+\infty, A) = Θ(k^{-1/2})$的充分必要条件。这完整解决了[WBNL26]中的问题。我们还给出了$d = o(k\log (n/k))$时实现大间隔的若干构造方案。最后，我们通过实验测试了InfoNCE和sigmoid损失函数在产生大间隔嵌入方面的表现，并展示了sigmoid损失的明显优势。