In this paper, we investigate the asymptotic behaviors of the extreme eigenvectors in a general spiked covariance matrix, where the dimension and sample size increase proportionally. We eliminate the restrictive assumption of the block diagonal structure in the population covariance matrix. Moreover, there is no requirement for the spiked eigenvalues and the 4th moment to be bounded. Specifically, we apply random matrix theory to derive the convergence and limiting distributions of certain projections of the extreme eigenvectors in a large sample covariance matrix within a generalized spiked population model. Furthermore, our techniques are robust and effective, even when spiked eigenvalues differ significantly in magnitude from nonspiked ones. Finally, we propose a powerful statistic for hypothesis testing for the eigenspaces of covariance matrices.

翻译：本文研究一般尖峰协方差矩阵中极端特征向量的渐近行为，其中维度和样本量成比例增加。我们消除了总体协方差矩阵中块对角结构的限制性假设。此外，尖峰特征值和四阶矩无需有界。具体而言，我们应用随机矩阵理论，在广义尖峰总体模型下推导大样本协方差矩阵中极端特征向量某些投影的收敛性和极限分布。进一步，即使尖峰特征值与非尖峰特征值的量级存在显著差异，我们的方法依然稳健有效。最后，我们提出了一种用于协方差矩阵特征空间假设检验的强力统计量。