We study the spectral properties of a class of random matrices of the form $S_n^{-} = n^{-1}(X_1 X_2^* - X_2 X_1^*)$ where $X_k = \Sigma^{1/2}Z_k$, for $k=1,2$, $Z_k$'s are independent $p\times n$ complex-valued random matrices, and $\Sigma$ is a $p\times p$ positive semi-definite matrix, independent of the $Z_k$'s. We assume that $Z_k$'s have independent entries with zero mean and unit variance. The skew-symmetric/skew-Hermitian matrix $S_n^{-}$ will be referred to as a random commutator matrix associated with the samples $X_1$ and $X_2$. We show that, when the dimension $p$ and sample size $n$ increase simultaneously, so that $p/n \to c \in (0,\infty)$, there exists a limiting spectral distribution (LSD) for $S_n^{-}$, supported on the imaginary axis, under the assumptions that the spectral distribution of $\Sigma$ converges weakly and the entries of $Z_k$'s have moments of sufficiently high order. This nonrandom LSD can be described through its Stieltjes transform, which satisfies a coupled Mar\v{c}enko-Pastur-type functional equations. In the special case when $\Sigma = I_p$, we show that the LSD of $S_n^{-}$ is a mixture of a degenerate distribution at zero (with positive mass if $c > 2$), and a continuous distribution with a symmetric density function supported on a compact interval on the imaginary axis. Moreover, we show that the companion matrix $S_n^{+} = \Sigma_n^\frac{1}{2}(Z_1Z_2^* + Z_2Z_1^*)\Sigma_n^\frac{1}{2}$, under identical assumptions, has an LSD supported on the real line, which can be similarly characterized.

翻译：我们研究一类随机矩阵的谱性质，其形式为 $S_n^{-} = n^{-1}(X_1 X_2^* - X_2 X_1^*)$，其中 $X_k = \Sigma^{1/2}Z_k$（$k=1,2$），$Z_k$ 是独立的 $p\times n$ 复值随机矩阵，$\Sigma$ 是 $p\times p$ 半正定矩阵且独立于 $Z_k$。我们假设 $Z_k$ 的元素相互独立，均值为零，方差为一。该斜对称/斜厄米特矩阵 $S_n^{-}$ 可视为与样本 $X_1$ 和 $X_2$ 相关联的随机交换子矩阵。我们证明，当维数 $p$ 与样本量 $n$ 同时增长且满足 $p/n \to c \in (0,\infty)$ 时，在 $\Sigma$ 的谱分布弱收敛且 $Z_k$ 的元素具有足够高阶矩的假设下，$S_n^{-}$ 存在一个支撑于虚轴上的极限谱分布。这一非随机的极限谱分布可通过其 Stieltjes 变换进行描述，该变换满足一组耦合的 Marčenko-Pastur 型函数方程。特别地，当 $\Sigma = I_p$ 时，我们证明 $S_n^{-}$ 的极限谱分布是退化分布（若 $c > 2$ 则在零点具有正质量）与一个支撑于虚轴紧区间上、具有对称密度函数的连续分布的混合。此外，我们证明在相同假设下，伴随矩阵 $S_n^{+} = \Sigma_n^\frac{1}{2}(Z_1Z_2^* + Z_2Z_1^*)\Sigma_n^\frac{1}{2}$ 存在一个支撑于实轴上的极限谱分布，该分布可类似地表征。