Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.

翻译：参数高效微调（PEFT）已成为在有限计算预算下适配大语言模型（LLM）的关键范式。然而，标准PEFT方法在多任务微调场景中常面临挑战：多样化的优化目标引发任务干扰，有限的参数预算导致表征能力不足。尽管近期方法通过引入混合专家（MoE）缓解这些问题，但这些方法主要在空间域中运作，可能带来结构冗余和参数开销。为突破这些局限，我们重新在谱域中定义适配问题。谱分析表明，不同任务呈现显著的频率能量分布差异，且大语言模型各层具有异质的频率敏感性。受此启发，我们提出FourierMoE，该模型将MoE架构与逆离散傅里叶变换（IDFT）相结合，实现频率感知适配。具体而言，FourierMoE采用频率自适应路由器将令牌分发至专精不同频段的专家。每个专家学习一组共轭对称复系数，在保留完整相位与振幅信息的同时，从理论上保证可无损重构为实值空间权重的IDFT重建。在28个基准测试、多种模型架构及参数规模上的广泛评估表明，FourierMoE在使用显著更少可训练参数的前提下，在单任务与多任务场景中均持续优于竞争基线。这些结果凸显了谱域专家适配作为大语言模型微调的有效且参数高效范式的潜力。