Fine-tuning large language models (LLMs) on a mixture of diverse datasets poses challenges due to data imbalance and heterogeneity. Existing methods often address these issues across datasets (globally) but overlook the imbalance and heterogeneity within individual datasets (locally), which limits their effectiveness. We introduce Hierarchical Balancing Optimization (HBO), a novel method that enables LLMs to autonomously adjust data allocation during fine-tuning both across datasets (globally) and within each individual dataset (locally). HBO employs a bilevel optimization strategy with two types of actors: a Global Actor, which balances data sampling across different subsets of the training mixture, and several Local Actors, which optimizes data usage within each subset based on difficulty levels. These actors are guided by reward functions derived from the LLM's training state, which measure learning progress and relative performance improvement. We evaluate HBO on three LLM backbones across nine diverse tasks in multilingual and multitask setups. Results show that HBO consistently outperforms existing baselines, achieving significant accuracy gains. Our in-depth analysis further demonstrates that both the global actor and local actors of HBO effectively adjust data usage during fine-tuning. HBO provides a comprehensive solution to the challenges of data imbalance and heterogeneity in LLM fine-tuning, enabling more effective training across diverse datasets.

翻译：在多样化的数据集混合体上对大语言模型进行微调时，由于数据不平衡和异质性而面临挑战。现有方法通常跨数据集（全局地）处理这些问题，但忽略了单个数据集内部（局部地）的不平衡和异质性，这限制了其有效性。我们提出了层次化平衡优化方法，这是一种新颖的方法，使大语言模型能够在微调过程中自主调整数据分配，既跨数据集（全局）又在每个单独的数据集内部（局部）。HBO采用一种双层优化策略，包含两种类型的执行器：一个全局执行器，用于平衡训练混合体中不同子集之间的数据采样；以及多个局部执行器，用于根据难度级别优化每个子集内部的数据使用。这些执行器由源自大语言模型训练状态的奖励函数指导，这些函数衡量学习进度和相对性能提升。我们在多语言和多任务设置中，针对九个不同任务，在三种大语言模型骨干上评估了HBO。结果表明，HBO始终优于现有基线方法，实现了显著的准确率提升。我们的深入分析进一步表明，HBO的全局执行器和局部执行器都能在微调过程中有效调整数据使用。HBO为大语言模型微调中数据不平衡和异质性的挑战提供了一个全面的解决方案，使得跨多样化数据集的训练更加有效。