We initiate the study of sub-linear sketching and streaming techniques for estimating the output size of common dictionary compressors such as Lempel-Ziv '77, the run-length Burrows-Wheeler transform, and grammar compression. To this end, we focus on a measure that has recently gained much attention in the information-theoretic community and which approximates up to a polylogarithmic multiplicative factor the output sizes of those compressors: the normalized substring complexity function $\delta$. We present a data sketch of $O(\epsilon^{-3}\log n + \epsilon^{-1}\log^2 n)$ words that allows computing a multiplicative $(1\pm \epsilon)$-approximation of $\delta$ with high probability, where $n$ is the string length. The sketches of two strings $S_1,S_2$ can be merged in $O(\epsilon^{-1}\log^2 n)$ time to yield the sketch of $\{S_1,S_2\}$, speeding up by orders of magnitude tasks such as the computation of all-pairs \emph{Normalized Compression Distances} (NCD). If random access is available on the input, our sketch can be updated in $O(\epsilon^{-1}\log^2 n)$ time for each character right-extension of the string. This yields a polylogarithmic-space algorithm for approximating $\delta$, improving exponentially over the working space of the state-of-the-art algorithms running in nearly-linear time. Motivated by the fact that random access is not always available on the input data, we then present a streaming algorithm computing our sketch in $O(\sqrt n \cdot \log n)$ working space and $O(\epsilon^{-1}\log^2 n)$ worst-case delay per character. We show that an implementation of our streaming algorithm can estimate {\delta} on a dataset of 189GB with a throughput of 203MB per minute while using only 5MB of RAM, and that our sketch speeds up the computation of all-pairs NCD distances by one order of magnitude, with applications to phylogenetic tree reconstruction.

翻译：我们首次系统研究了用于估计常见词典压缩器（如Lempel-Ziv '77、游程Burrows-Wheeler变换及文法压缩）输出规模的亚线性草图与流式技术。为此，我们聚焦于信息论领域近年备受关注的度量标准——归一化子串复杂度函数$\delta$，该函数能以多对数乘法因子近似上述压缩器的输出规模。我们提出一种包含$O(\epsilon^{-3}\log n + \epsilon^{-1}\log^2 n)$个单词的数据草图，能以高概率计算$\delta$的乘法$(1\pm\epsilon)$近似值（其中$n$为字符串长度）。两个字符串$S_1,S_2$的草图可在$O(\epsilon^{-1}\log^2 n)$时间内合并为$\{S_1,S_2\}$的草图，从而将全对归一化压缩距离（NCD）计算等任务的速度提升数个数量级。若输入支持随机访问，对于字符串的每次右扩字符，我们的草图可在$O(\epsilon^{-1}\log^2 n)$时间内完成更新。这实现了对$\delta$的多对数空间近似算法，比当前最优近线性时间算法的工作空间指数级优化。考虑到输入数据并非总能支持随机访问，我们进一步提出一种流式算法，该算法在$O(\sqrt n \cdot \log n)$工作空间和每字符$O(\epsilon^{-1}\log^2 n)$最坏情况延迟下完成草图计算。实验表明，该流式算法实现对189GB数据集的$\delta$估算时，吞吐量达203MB/分钟，仅需5MB内存；同时，我们的草图将全对NCD距离计算速度提升一个数量级，可应用于系统发育树重建任务。