Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present challenges, such as managing inter-document dependencies, redundancy, and incoherent structures. We introduce MDCure, a scalable and effective fine-tuning pipeline to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human annotated data. MDCure is based on generation of high-quality synthetic MD instruction data from sets of related articles via targeted prompts. We further introduce MDCureRM, a multi-objective reward model which filters generated data based on their training utility for MD settings. With MDCure, we fine-tune a variety of LLMs, from the FlanT5, Qwen2, and LLAMA3.1 model families, up to 70B parameters in size. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks show MDCure consistently improves performance over pre-trained baselines and over corresponding base models by up to 75.5%. Our code, datasets, and models are available at https://github.com/yale-nlp/MDCure.

翻译：多文档（MD）处理对于大型语言模型（LLM）处理现实世界任务（如跨大规模文档集的摘要和问答）至关重要。尽管LLM在处理长输入方面已有所改进，但多文档上下文仍面临诸多挑战，例如管理文档间依赖关系、冗余信息以及不连贯的结构。我们提出了MDCure，一种可扩展且高效的微调流水线，旨在增强LLM的多文档处理能力，而无需承担预训练的计算成本或依赖人工标注数据。MDCure基于通过针对性提示从相关文章集合生成高质量合成多文档指令数据。我们进一步引入了MDCureRM，一种多目标奖励模型，可根据生成数据在多文档场景下的训练效用进行筛选。利用MDCure，我们对多种LLM进行了微调，涵盖FlanT5、Qwen2和LLAMA3.1模型系列，参数规模最高达700亿。在涵盖多种任务的多文档及长上下文基准测试上进行广泛评估，结果表明MDCure持续提升模型性能，相较于预训练基线及对应基础模型，最高提升幅度达75.5%。我们的代码、数据集和模型已在https://github.com/yale-nlp/MDCure公开。