We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-error.

翻译：我们提出了NerVE，一个统一的本征谱框架，用于理解大语言模型（LLMs）中的前馈网络（FFNs）如何在高维潜在空间中组织和调控信息流。尽管FFNs占据了参数预算的主要部分，但其高维动力学机制仍鲜为人知。NerVE通过四种互补的度量指标——谱熵（离散度）、参与比（有效维度）、本征值早期富集（顶部权重集中度）和Jensen-Shannon散度（分布偏移）——对特征谱动力学进行轻量级、内存高效的追踪，从而弥补了这一空白。我们的核心见解是：FFN的非线性特性会跨本征模重新注入方差，从根本上支配着潜在维度的利用；而优化器的几何结构则强烈调节着这种方差重新注入的程度。我们在不同模型规模、多样化的架构和优化器配置中验证了NerVE，每种配置都独特地塑造了FFN的动力学：归一化方案控制方差流动；FFN权重几何约束潜在空间；位置编码和激活函数调控信息流；优化器选择在深度维度上重新分配有效容量。在这些不同设置中，NerVE始终能提取出稳定的谱特征，这些特征与模型的泛化能力相关，并对设计选择做出可预测的响应。该框架可推广至Transformer之外的MLP-Mixer架构，为超越试错法的架构与优化器选择提供了可操作的见解。