We initiate the study of sub-linear sketching and streaming techniques for estimating the output size of common dictionary compressors such as Lempel-Ziv '77, the run-length Burrows-Wheeler transform, and grammar compression. To this end, we focus on a measure that has recently gained much attention in the information-theoretic community and which approximates up to a polylogarithmic multiplicative factor the output sizes of those compressors: the normalized substring complexity function $\delta$. As a matter of fact, $\delta$ itself is a very accurate measure of compressibility: it is monotone under concatenation, invariant under reversals and alphabet permutations, sub-additive, and asymptotically tight (in terms of worst-case entropy) for representing strings, up to polylogarithmic factors. We present a data sketch of $O(\epsilon^{-3}\log n + \epsilon^{-1}\log^2 n)$ words that allows computing a multiplicative $(1\pm \epsilon)$-approximation of $\delta$ with high probability, where $n$ is the string length. The sketches of two strings $S_1,S_2$ can be merged in $O(\epsilon^{-1}\log^2 n)$ time to yield the sketch of $\{S_1,S_2\}$, speeding up by orders of magnitude tasks such as the computation of all-pairs \emph{Normalized Compression Distances} (NCD). If random access is available on the input, our sketch can be updated in $O(\epsilon^{-1}\log^2 n)$ time for each character right-extension of the string. This yields a polylogarithmic-space algorithm for approximating $\delta$, improving exponentially over the working space of the state-of-the-art algorithms running in nearly-linear time. Motivated by the fact that random access is not always available on the input data, we then present a streaming algorithm computing our sketch in $O(\sqrt n \cdot \log n)$ working space and $O(\epsilon^{-1}\log^2 n)$ worst-case delay per character.

翻译：我们首次研究利用次线性草图与流式技术来估计常见字典压缩器（如Lempel-Ziv '77、游程编码Burrows-Wheeler变换及文法压缩）输出规模的问题。为此，我们聚焦于信息论界近期备受关注的一个度量指标——归一化子串复杂度函数$\delta$，该指标可在多项式对数因子近似范围内逼近上述压缩器的输出规模。事实上，$\delta$本身即为一种高度精确的可压缩性度量：它在字符串连接下具有单调性，对反转和字母置换保持不变性，满足次可加性，且在最坏情况熵意义上（最多相差多项式对数因子）对字符串表示具有渐近紧致性。我们提出一种由$O(\epsilon^{-3}\log n + \epsilon^{-1}\log^2 n)$个词构成的数据草图，能够以高概率计算$\delta$的乘法$(1\pm \epsilon)$近似值，其中$n$为字符串长度。两个字符串$S_1,S_2$的草图可在$O(\epsilon^{-1}\log^2 n)$时间内合并得到$\{S_1,S_2\}$的草图，从而将全对归一化压缩距离（NCD）计算等任务的速度提升数个数量级。若输入支持随机访问，我们的草图可在每次字符串向右扩展一个字符时以$O(\epsilon^{-1}\log^2 n)$时间完成更新。这给出了一个用于近似$\delta$的多项式对数空间算法，其工作空间较当前最优的近线性时间复杂度算法呈指数级改进。鉴于输入数据并非总支持随机访问，我们进一步提出一种流式算法：该算法以$O(\sqrt n \cdot \log n)$工作空间和每个字符$O(\epsilon^{-1}\log^2 n)$最坏情况延迟完成草图计算。