Weight-Decomposed Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction, but its forward pass requires the row-wise norm of W + sBA, a computation that every major framework we surveyed implements by materializing the dense [d_out, d_in] product BA. At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16, making high-rank DoRA costly and often infeasible on common single-GPU setups once hundreds of adapted modules and checkpointing are involved. We present two systems contributions. A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r^2) intermediates, eliminating the dense product. Fused Triton kernels collapse the four-kernel DoRA composition into a single pass, reducing memory traffic by about 4x and using a numerically stable form that avoids catastrophic cancellation in the near-unity rescaling regime where magnitude scales concentrate in practice. Across six 8-32B vision-language models (VLMs) on three NVIDIA GPUs (RTX 6000 PRO, H200, B200) at r = 384 in bf16, the fused implementation is 1.5-2.0x faster than Hugging Face PEFT's DoRA implementation for inference and 1.5-1.9x faster for gradient computation (optimizer step excluded), with up to 7 GB lower peak VRAM. Microbenchmarks on six GPUs spanning four architecture generations (L40S, A100, RTX 6000 PRO, H200, B200, B300) confirm 1.5-2.7x compose-kernel speedup. Final-logit cosine similarity exceeds 0.9999 across all model/GPU pairs, and multi-seed training curves match within 7.1 x 10^-4 mean per-step loss delta over 2000 steps.

翻译：权重分解低秩适配（DoRA）通过将权重幅值与方向解耦扩展了LoRA，但其前向计算需要计算W + sBA的行范数——我们调研的所有主流框架均通过显式物化稠密[d_out, d_in]矩阵乘积BA来实现该计算。当d_in=8192且秩r=384时，单个模块的范数计算在bf16精度下需要约512 MB的临时工作内存，这使得高秩DoRA在涉及数百个适配模块和梯度检查点时的常见单GPU场景中成本高昂且往往不可行。我们提出两项系统贡献：分解范数将平方范数分解为基项、交叉项和格拉姆项，通过O(d_out r + r^2)中间变量计算，消除了稠密矩阵乘积；融合Triton核将四核DoRA组合压缩为单次传递，内存流量减少约4倍，并采用数值稳定的形式避免接近1的缩放区域（实际应用中幅值缩放系数集中区域）出现灾难性抵消。在三种NVIDIA GPU（RTX 6000 PRO、H200、B200）上对六个8-32B视觉语言模型（VLM）进行r=384的bf16测试，融合实现的推理速度较Hugging Face PEFT的DoRA实现快1.5-2.0倍，梯度计算（不含优化器步骤）快1.5-1.9倍，峰值显存降低高达7 GB。横跨四代架构（L40S、A100、RTX 6000 PRO、H200、B200、B300）的六款GPU微基准测试证实组合核加速比达1.5-2.7倍。所有模型/GPU对的最终logit余弦相似度超过0.9999，多随机种子的2000步训练曲线均方每步损失差异在7.1×10^-4以内。