We build a benchmark to evaluate large language models (LLMs) for source code migration tasks, specifically upgrading functions from Java 8 to Java 11. We first collected a dataset of function pairs from open-source repositories, but limitations in data quality led us to construct a refined dataset covering eight categories of deprecated APIs. Using this dataset, the Mistral Codestral model was evaluated with CodeBLEU and keyword-based metrics to measure lexical and semantic similarity as well as migration correctness. Results show that the evaluated model (Mistral Codestral) can handle trivial one-to-one API substitutions with moderate success, achieving identical migrations in 11.11% of the cases, but it struggles with more complex migrations such as CORBA or JAX-WS. These findings suggest Mistral Codestral can partially reduce developer effort by automating repetitive migration tasks but cannot yet replace humans within the scope of the JMigBench benchmark. The benchmark and analysis provide a foundation for future work on expanding datasets, refining prompting strategies, and improving migration performance across different LLMs.

翻译：我们构建了一个基准，用于评估大型语言模型在源代码迁移任务上的性能，具体任务是将函数从 Java 8 升级到 Java 11。我们首先从开源仓库收集了一个函数对数据集，但由于数据质量的限制，我们构建了一个涵盖八类已弃用 API 的精细化数据集。利用该数据集，我们使用 CodeBLEU 和基于关键词的指标评估了 Mistral Codestral 模型，以衡量其词汇与语义相似度以及迁移正确性。结果表明，所评估的模型（Mistral Codestral）能够以中等成功率处理简单的一对一 API 替换，在 11.11% 的案例中实现了完全一致的迁移，但在处理更复杂的迁移（如 CORBA 或 JAX-WS）时仍存在困难。这些发现表明，Mistral Codestral 可以通过自动化重复性迁移任务来部分减少开发人员的工作量，但在 JMigBench 基准的范围内尚无法完全替代人工。该基准与分析为未来扩展数据集、优化提示策略以及提升不同大型语言模型迁移性能的研究奠定了基础。