The challenge of estimating similarity between sets has been a significant concern in data science, finding diverse applications across various domains. However, previous approaches, such as MinHash, have predominantly centered around hashing techniques, which are well-suited for sets but less naturally adaptable to multisets, a common occurrence in scenarios like network streams and text data. Moreover, with the increasing prevalence of data arriving in streaming patterns, many existing methods struggle to handle cases where set items are presented in a continuous stream. Consequently, our focus in this paper is on the challenging scenario of multisets with item streams. To address this, we propose SimiSketch, a sketching algorithm designed to tackle this specific problem. The paper begins by presenting two simpler versions that employ intuitive sketches for similarity estimation. Subsequently, we formally introduce SimiSketch and leverage SALSA to enhance accuracy. To validate our algorithms, we conduct extensive testing on synthetic datasets, real-world network traffic, and text articles. Our experiment shows that compared with the state-of-the-art, SimiSketch can improve the accuracy by up to 42 times, and increase the throughput by up to 360 times. The complete source code is open-sourced and available on GitHub for reference.

翻译：在数据科学领域，集合相似度估计一直是一个重要课题，并在诸多领域具有广泛应用。然而，现有方法（如MinHash）主要基于哈希技术，这类方法虽适用于普通集合，却难以自然扩展到多重集——这种在网络流和文本数据等场景中普遍存在的数据结构。此外，随着流式数据模式的日益普及，许多现有方法难以处理集合元素以连续流形式到达的情况。因此，本文聚焦于元素流形式的多重集这一挑战性场景。为此，我们提出SimiSketch——一种专门针对该问题的草图算法。本文首先给出两个采用直观草图进行相似度估计的简化版本，随后正式引入SimiSketch，并利用SALSA技术提升估计精度。为验证算法有效性，我们在合成数据集、真实网络流量和文本数据上进行了广泛测试。实验表明，相较于现有最优方法，SimiSketch可将估计精度提升高达42倍，并将吞吐量提高达360倍。完整源代码已在GitHub开源以供参考。