Linear layers hold most of a transformer's parameters. We replace each linear layer with one that stores $K$ out of $mn$ two-dimensional DCT coefficients per weight matrix and reconstructs the full matrix through an inverse DCT at every forward pass; the $K$ coefficients are the trainable parameters. A 4-layer, 128-dim transformer trained from scratch on character-level Shakespeare reaches validation loss $1.604$ at $K = mn/2$, against $1.580$ for a standard dense baseline -- a gap of $+0.024$ at half the trainable parameter count, within the terminal-epoch variation of the dense run. A rank-48 LoRA factorization at the same trainable parameter count reaches only $1.801$ ($+0.221$). The structural advantage of sparse-coefficient over low-rank parameterizations at matched $K$ is qualitative. We identify rank flexibility as the mechanism. A random orthonormal basis matches the DCT within noise at $K = mn/2$, and a compression sweep through $K = mn/10$ and $K = mn/20$ shows that subspaces that can host high-rank matrices keep the loss low, while subspaces that flatten into a low-rank block (zigzag-selection variants) converge onto the observed stable rank \emph{and} the loss line of the rank-48 LoRA reference in lock-step. Among these orthonormal bases, the DCT is preferred because its separable fast transform admits a fused reconstruction kernel: the materialized weight matrix never leaves on-chip memory, so the parameter saving translates into a bandwidth saving as well.

翻译：线性层占据了Transformer大部分参数。我们将每个线性层替换为：针对每个权重矩阵，仅存储$mn$个二维DCT系数中的$K$个，并在每次前向传播时通过逆DCT重建完整矩阵；这$K$个系数即为可训练参数。在字符级莎士比亚数据集上从零训练的四层128维Transformer，在$K = mn/2$时达到验证损失$1.604$，而标准稠密基线为$1.580$——即在可训练参数数量减半的条件下差距仅为$+0.024$，且此差距在稠密运行终期迭代的波动范围内。相同可训练参数数量下的秩-48 LoRA分解仅达到$1.801$（$+0.221$）。在相等$K$条件下，稀疏系数参数化相比低秩参数化具有结构性优势。我们将秩灵活性识别为其中的机制。当$K = mn/2$时，随机标准正交基在噪声范围内与DCT性能相当；而通过$K = mn/10$至$K = mn/20$的压缩扫描表明：能够容纳高秩矩阵的子空间可保持低损失，而退化为低秩块的子空间（锯齿形选择变体）会同步收敛到观测到的稳定秩与秩-48 LoRA参考的损失线。在这些标准正交基中，DCT因具备可分离快速变换而备受青睐——其实体化权重矩阵始终驻留于片上存储器，从而参数节约可直接转化为带宽节约。