Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases.

翻译：通过参数高效微调（PEFT）技术（例如LoRA）的引入，使大型语言模型（LLM）适应新任务的微调过程变得更加高效。然而，与全参数微调相比，这些方法通常表现欠佳，尤其是在涉及复杂数据集的场景中。这一问题在复杂领域中变得更为突出，凸显了对能够实现更优性能的改进型PEFT方法的需求。通过一系列实验，我们发现了两个关键见解，揭示了LoRA在训练和参数效率方面的不足。基于这些见解，我们开发了HydraLoRA，这是一种具有非对称结构的LoRA框架，无需领域专业知识。我们的实验表明，HydraLoRA优于其他PEFT方法，甚至优于那些在训练和推理阶段依赖领域知识的方法。