Large-scale embedding-based retrieval (EBR) is the cornerstone of search-related industrial applications. Given a user query, the system of EBR aims to identify relevant information from a large corpus of documents that may be tens or hundreds of billions in size. The storage and computation turn out to be expensive and inefficient with massive documents and high concurrent queries, making it difficult to further scale up. To tackle the challenge, we propose a binary embedding-based retrieval (BEBR) engine equipped with a recurrent binarization algorithm that enables customized bits per dimension. Specifically, we compress the full-precision query and document embeddings, formulated as float vectors in general, into a composition of multiple binary vectors using a lightweight transformation model with residual multilayer perception (MLP) blocks. We can therefore tailor the number of bits for different applications to trade off accuracy loss and cost savings. Importantly, we enable task-agnostic efficient training of the binarization model using a new embedding-to-embedding strategy. We also exploit the compatible training of binary embeddings so that the BEBR engine can support indexing among multiple embedding versions within a unified system. To further realize efficient search, we propose Symmetric Distance Calculation (SDC) to achieve lower response time than Hamming codes. We successfully employed the introduced BEBR to Tencent products, including Sogou, Tencent Video, QQ World, etc. The binarization algorithm can be seamlessly generalized to various tasks with multiple modalities. Extensive experiments on offline benchmarks and online A/B tests demonstrate the efficiency and effectiveness of our method, significantly saving 30%~50% index costs with almost no loss of accuracy at the system level.

翻译：大规模基于嵌入的检索（EBR）是搜索相关工业应用的基石。给定用户查询，EBR系统旨在从可能包含数百亿规模的大型文档语料库中识别相关信息。面对海量文档和高并发查询，存储和计算成本变得昂贵且低效，阻碍了系统的进一步扩展。为解决这一挑战，我们提出了一种配备循环二值化算法的二进制嵌入检索（BEBR）引擎，该算法支持自定义每维度位数。具体而言，我们通过使用带有残差多层感知器（MLP）模块的轻量级变换模型，将通常表示为浮点向量的全精度查询和文档嵌入压缩为多个二进制向量的组合。因此，我们可以针对不同应用定制位数，以权衡精度损失与成本节约。重要的是，我们采用一种新的嵌入到嵌入策略，实现了二值化模型的任务无关高效训练。我们还探索了二进制嵌入的兼容训练，使BEBR引擎能在统一系统中支持多版本嵌入的索引。为进一步实现高效搜索，我们提出了对称距离计算（SDC），其响应时间低于汉明码。我们成功将所提出的BEBR部署于腾讯产品中，包括搜狗、腾讯视频、QQ世界等。该二值化算法可无缝推广至多模态的各类任务。在离线基准测试和在线A/B测试上的大量实验证明了我们方法的效率和有效性，在系统层面几乎无损精度的情况下，显著节省了30%~50%的索引成本。