With the rapid advancement of powerful large language models (LLMs) in recent years, a wide range of software engineering tasks can now be addressed using LLMs, significantly enhancing productivity and scalability. Numerous benchmark datasets have been developed to evaluate the coding capabilities of these models, while they primarily focus on code generation and issue-resolution tasks. In contrast, we introduce a new coding benchmark MigrationBench with a distinct focus: code migration. MigrationBench aims to serve as a comprehensive benchmark for migration from Java 8 to the latest long-term support (LTS) versions (Java 17, 21), including a full dataset and its subset selected with 5,102 and 300 repositories respectively. selected is a representative subset curated for complexity and difficulty, offering a versatile resource to support research in the field of code migration. Additionally, we provide a comprehensive evaluation framework to facilitate rigorous and standardized assessment of LLMs on this challenging task. We further propose an agentic framework and demonstrate that LLMs can effectively tackle repository-level code migration to Java 17. For the selected subset with Claude-4.5-Sonnet, our agentic framework achieves 71.67% and 53.33% success rate (pass@1) for minimal and maximal migration respectively. The dataset and evaluation source code are available at: https://huggingface.co/collections/AmazonScience/migrationbench and https://github.com/amazon-science/MigrationBench respectively.

翻译：摘要：近年来，随着强大大型语言模型（LLMs）的快速发展，如今大量软件工程任务可借助LLMs解决，显著提升了生产力和可扩展性。现有众多基准数据集虽被开发用于评估此类模型的编码能力，但主要聚焦于代码生成和问题解决任务。与之相对，我们提出了一个具有独特关注点的新型编码基准测试——代码迁移领域的MigrationBench。该基准旨在为从Java 8向最新长期支持（LTS）版本（Java 17、21）的迁移提供全面评估，其完整数据集及子集分别包含5102个和300个代码仓库。其中，所选子集是基于复杂度和难度精心筛选的代表性样本，可为代码迁移领域的研究提供多功能资源。此外，我们构建了综合性评估框架，以促进对该挑战性任务中LLMs进行严格、标准化的评估。进一步，我们提出基于智能体的框架，并证明LLMs可有效处理向Java 17的仓库级代码迁移。以Claude-4.5-Sonnet为例，我们的智能体框架在所选子集上的最小迁移和最大迁移成功率（pass@1）分别达到71.67%和53.33%。数据集与评估源代码已分别发布于：https://huggingface.co/collections/AmazonScience/migrationbench 和 https://github.com/amazon-science/MigrationBench。