Large language models (LLMs) excel in open domains but struggle in specialized settings with limited data and evolving knowledge. Existing domain adaptation practices rely heavily on manual trial-and-error processes, incur significant hyperparameter complexity, and are highly sensitive to data and user preferences, all under the high cost of LLM training. Moreover, the interactions and transferability of hyperparameter choices across models/domains remain poorly understood, making adaptation gains uncertain even with substantial effort. To solve these challenges, we present AutoAdapt, a novel end-to-end automated framework for efficient and reliable LLM domain adaptation. AutoAdapt leverages curated knowledge bases from literature and open-source resources to reduce expert intervention. To narrow the search space, we design a novel multi-agent debating system in which proposal and critic agents iteratively interact to align user intent and incorporate data signals and best practices into the planning process. To optimize hyperparameters under tight budgets, we propose AutoRefine, a novel LLM-based surrogate that replaces costly black-box search. Across 10 tasks, AutoAdapt achieves a 25% average relative accuracy improvement over state-of-the-art Automated Machine Learning baselines with minimal overhead.

翻译：大语言模型（LLMs）在开放领域表现卓越，但在数据有限且知识持续演化的专业场景中面临挑战。现有的领域自适应方法严重依赖人工试错过程，存在显著的超参数复杂性，对数据和用户偏好高度敏感，且均需承担高昂的LLM训练成本。此外，超参数选择在不同模型/领域间的交互作用与可迁移性仍不明确，导致即使投入大量精力，自适应效果的提升仍存在不确定性。为解决这些难题，本文提出AutoAdapt——一种新颖的端到端自动化框架，旨在实现高效可靠的大语言模型领域自适应。AutoAdapt利用文献与开源资源构建的知识库来减少专家干预。为压缩搜索空间，我们设计了一种创新的多智能体辩论系统，其中提案智能体与批评智能体通过迭代交互对齐用户意图，并将数据信号与最佳实践纳入规划流程。为在严格预算下优化超参数，我们提出了AutoRefine——一种基于LLM的新型代理模型，用以替代昂贵的黑盒搜索。在10项任务上的实验表明，AutoAdapt以最小开销实现了相较于最先进的自动化机器学习基线方法平均25%的相对准确率提升。