Long-rollout causal video diffusion has converged on a fixed-size sliding-window KV cache, with recent progress innovating within this layout by changing which tokens occupy the window or how their positions are encoded. The per-head KV layout itself, a dominant contributor to streaming memory and latency, has been mostly left unchanged. In this paper, we present the first study of Multi-Head Latent Attention (MLA) in video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7% at every cached layer. We further investigate why MLA succeeds in video diffusion even though the spectral assumption often used to motivate it in language models does not hold: pretrained video attention is not low-rank, with 99%-energy effective rank far above any practical latent dimension. VideoMLA retains quality at compression ratios where direct spectral approximation would predict large reconstruction error. We show that the MLA bottleneck, rather than the pretrained spectrum, determines the effective rank: both spectral and random initialization occupy nearly the full rank budget from initialization, and training preserves this budget while adapting within it. On VBench, VideoMLA matches short-horizon streaming video diffusion baselines, achieves the best overall score at long horizons among evaluated methods, and improves throughput by 1.23x on a single B200.

翻译：长序列因果视频扩散已形成固定大小的滑动窗口KV缓存，近期进展通过改变窗口内保留的令牌或位置编码方式在该框架内进行创新。然而，作为流式内存和延迟主要贡献者的每头KV布局本身基本保持不变。本文首次提出将多头潜在注意力（MLA）用于视频扩散。VideoMLA将每头的键和值替换为共享的低秩内容潜在表示和共享的解耦3D-RoPE位置键，使得每缓存层的每令牌KV内存减少92.7%。我们进一步研究了MLA在视频扩散中成功的原因，尽管语言模型中常用的频谱假设（即预训练视频注意力非低秩，99%能量有效秩远超任何实际潜在维度）在此不成立。VideoMLA在压缩比下保持质量，而直接频谱近似在该比下预期会产生较大重建误差。我们证明，决定有效秩的是MLA瓶颈而非预训练频谱：谱初始化和随机初始化均从初始化起占据近乎完整的秩预算，训练在保持该预算的同时在其中进行自适应。在VBench上，VideoMLA匹敌短时窗流式视频扩散基线，在长时窗评估方法中取得最佳整体得分，并在单块B200上将吞吐量提升至1.23倍。