Visualization of extremely large datasets in static or dynamic form is a huge challenge because most traditional methods cannot deal with big data problems. A new visualization method for big data is proposed based on Projection Pursuit, Guided Tour and Data Nuggets methods, that will help display interesting hidden structures such as clusters, outliers, and other nonlinear structures in big data. The Guided Tour is a dynamic graphical tool for high-dimensional data combining Projection Pursuit and Grand Tour methods. It displays a dynamic sequence of low-dimensional projections obtained by using Projection Pursuit (PP) index functions to navigate the data space. Different PP indices have been developed to detect interesting structures of multivariate data but there are computational problems for big data using the original guided tour with these indices. A new PP index is developed to be computable for big data, with the help of a data compression method called Data Nuggets that reduces large datasets while maintaining the original data structure. Simulation studies are conducted and a real large dataset is used to illustrate the proposed methodology. Static and dynamic graphical tools for big data can be developed based on the proposed PP index to detect nonlinear structures.
翻译:静态或动态形式下超大规模数据集的可视化是一项巨大挑战,因为大多数传统方法无法处理大数据问题。提出一种基于投影寻踪、导向游走和数据块方法的大数据可视化新方法,有助于展示大数据中诸如聚类、离群点及其他非线性结构等有趣的隐藏结构。导向游走是一种结合投影寻踪与全局游走方法的高维数据动态图形工具,通过使用投影寻踪指标函数导航数据空间,展示低维投影的动态序列。目前已开发多种投影寻踪指标用于检测多变量数据的有趣结构,但在大数据场景下使用原始导向游走配合这些指标存在计算问题。本文借助名为"数据块"的数据压缩方法(该方法在保持原始数据结构的同时缩减大数据集规模),开发了一种适用于大数据计算的新型投影寻踪指标。通过模拟研究及真实大规模数据集验证了所提方法,可基于该投影寻踪指标开发大数据的静态与动态图形工具以检测非线性结构。