Mainframe operating systems, despite their inception in the 1940s, continue to support critical sectors like finance and government. However, these systems are often viewed as outdated, requiring extensive maintenance and modernization. Addressing this challenge necessitates innovative tools that can understand and interact with legacy codebases. To this end, we introduce XMainframe, a state-of-the-art large language model (LLM) specifically designed with knowledge of mainframe legacy systems and COBOL codebases. Our solution involves the creation of an extensive data collection pipeline to produce high-quality training datasets, enhancing XMainframe's performance in this specialized domain. Additionally, we present MainframeBench, a comprehensive benchmark for assessing mainframe knowledge, including multiple-choice questions, question answering, and COBOL code summarization. Our empirical evaluations demonstrate that XMainframe consistently outperforms existing state-of-the-art LLMs across these tasks. Specifically, XMainframe achieves 30% higher accuracy than DeepSeek-Coder on multiple-choice questions, doubles the BLEU score of Mixtral-Instruct 8x7B on question answering, and scores six times higher than GPT-3.5 on COBOL summarization. Our work highlights the potential of XMainframe to drive significant advancements in managing and modernizing legacy systems, thereby enhancing productivity and saving time for software developers.

翻译：大型机操作系统尽管诞生于20世纪40年代，至今仍支撑着金融和政府等关键领域。然而，这些系统常被视为过时技术，需要大量维护与现代化改造。应对这一挑战需要能够理解并处理遗留代码库的创新工具。为此，我们推出XMainframe——一种专门针对大型机遗留系统与COBOL代码库知识设计的最先进大型语言模型（LLM）。我们的解决方案通过构建大规模数据收集流水线来生成高质量训练数据集，从而提升XMainframe在该专业领域的性能。此外，我们提出了MainframeBench——一个用于评估大型机知识的综合性基准测试，包含多项选择题、问答任务及COBOL代码摘要生成。实验评估表明，XMainframe在所有任务中均持续超越现有最先进的LLM。具体而言，在多项选择题上XMainframe比DeepSeek-Coder准确率提升30%，在问答任务上其BLEU分数是Mixtral-Instruct 8x7B的两倍，在COBOL摘要任务中的得分更是GPT-3.5的六倍。本项工作彰显了XMainframe在推动遗留系统管理与现代化方面实现重大进展的潜力，从而提升软件开发者的工作效率并节约时间成本。