Deep clustering, a method for partitioning complex, high-dimensional data using deep neural networks, presents unique evaluation challenges. Traditional clustering validation measures, designed for low-dimensional spaces, are problematic for deep clustering, which involves projecting data into lower-dimensional embeddings before partitioning. Two key issues are identified: 1) the curse of dimensionality when applying these measures to raw data, and 2) the unreliable comparison of clustering results across different embedding spaces stemming from variations in training procedures and parameter settings in different clustering models. This paper addresses these challenges in evaluating clustering quality in deep learning. We present a theoretical framework to highlight ineffectiveness arising from using internal validation measures on raw and embedded data and propose a systematic approach to applying clustering validity indices in deep clustering contexts. Experiments show that this framework aligns better with external validation measures, effectively reducing the misguidance from the improper use of clustering validity indices in deep learning.

翻译：深度聚类是一种利用深度神经网络对复杂高维数据进行划分的方法，其评估面临独特挑战。传统聚类验证指标针对低维空间设计，在深度聚类中存在两个关键问题：1) 将这些指标应用于原始数据时的维数灾难；2) 不同聚类模型因训练流程与参数设置差异导致嵌入空间各异，使得跨空间聚类结果不可靠比较。本文针对深度学习中的聚类质量评估难题，提出了一个理论框架以揭示在原始数据和嵌入数据上使用内部验证指标的低效性，并系统性地提出在深度聚类场景中应用聚类有效性指标的方法。实验表明，该框架与外部验证指标的契合度更高，能有效降低深度学习中对聚类有效性指标不当使用所带来的误导。