Large language models have transformed code generation, enabling unprecedented automation in software development. As mobile ecosystems evolve, HarmonyOS has emerged as a critical platform requiring robust development tools. Software development for the HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript. Despite its growing importance, the ecosystem lacks robust tools for automated code repair, primarily due to the absence of a high-quality benchmark for evaluation. To address this gap, we present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction. It provides the first comprehensive benchmark specifically designed for ArkTS automated program repair. We constructed this benchmark by mining issues from a large-scale official Huawei repository containing over 400 independent ArkTS applications. Through a rigorous multi-stage filtering process, we curated 502 reproducible issues. To ensure testability, we employed a novel LLM-based test generation and voting mechanism involving Claude and other models. Furthermore, we standardized problem statements to facilitate fair evaluation. Finally, we evaluated four state-of-the-art Large Language Models (LLMs) on our benchmark using a retrieval-augmented repair workflow. Our results highlight the current capabilities and limitations of LLMs in repairing ArkTS code, paving the way for future research in this low-resource language domain.

翻译：大型语言模型已彻底改变代码生成领域，为软件开发带来前所未有的自动化水平。随着移动生态系统的演进，HarmonyOS已成为需要强大开发工具的关键平台。HarmonyOS生态系统的软件开发高度依赖ArkTS——一种TypeScript的静态类型扩展。尽管其重要性日益增长，该生态系统仍缺乏强大的自动化代码修复工具，主要原因是缺乏高质量的评估基准。为填补这一空白，我们提出ArkEval：一个用于ArkTS自动化修复工作流评估与基准构建的统一框架。它提供了首个专为ArkTS自动化程序修复设计的综合性基准。我们通过从包含400多个独立ArkTS应用的大规模华为官方代码库中挖掘问题，构建了该基准。经过严格的多阶段筛选流程，我们最终筛选出502个可复现的问题。为确保可测试性，我们采用了一种基于LLM的新型测试生成与投票机制，涉及Claude及其他模型。此外，我们标准化了问题描述以促进公平评估。最后，我们使用检索增强的修复工作流在基准上评估了四种最先进的大型语言模型。我们的结果揭示了LLM在修复ArkTS代码方面的当前能力与局限，为这一低资源语言领域的未来研究铺平了道路。