When faced with new data, we often conduct a cluster analysis to obtain a better understanding of the data's structure and the archetypical samples present in the data. However, the increases in data complexity and dimensionality have made this step very tricky. The large proportion of noise in high-dimensional data blurs patterns and trends, making clusters difficult to distinguish. As such, cluster-discovery tools and cluster-verification tools must be adapted to address the difficulties of high-dimensional data. Nonlinear dimension reduction is a step in the right direction, but even these methods are known to produce false structures, especially when mishandled. A common phenomenon that often goes undetected by the untrained eye is over-clustering of the data. In continuation of these efforts, we developed new cluster verification techniques, including visual assessments and a hypothesis test, that help analysts distinguish false clusters and better interpret their high-dimensional clustering results. For ease of use, these new methods are provided in an interactive toolbox available via R package DRTool.

翻译：暂无翻译