In the big data era, the key feature that each algorithm needs to have is the possibility of efficiently running in parallel in a distributed environment. The popular Silhouette metric to evaluate the quality of a clustering, unfortunately, does not have this property and has a quadratic computational complexity with respect to the size of the input dataset. For this reason, its execution has been hindered in big data scenarios, where clustering had to be evaluated otherwise. To fill this gap, in this paper we introduce the first algorithm that computes the Silhouette metric with linear complexity and can easily execute in parallel in a distributed environment. Its implementation is freely available in the Apache Spark ML library.

翻译：在大数据时代，算法必须具备的一项关键特性是能够在分布式环境中高效并行运行。然而，用于评估聚类质量的流行指标——轮廓系数（Silhouette）并不具备这一特性，并且其计算复杂度与输入数据集的大小呈二次方关系。因此，在大数据场景中，该指标的执行受到阻碍，聚类质量不得不采用其他方式评估。为弥补这一空白，本文首次提出了一种能够在线性复杂度下计算轮廓系数并易于在分布式环境中并行执行的算法。该算法已在Apache Spark ML库中开源实现。