Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming.

翻译：近年来，大语言模型（LLMs）在代码生成方面展现出强大潜力，但其在量子计算领域的有效性仍未得到充分探索。本文利用量子黑客松（QHack）的真实挑战任务，对基于PennyLane的量子代码生成LLMs进行了基准测试。我们提出了QHackBench——一个源自QHack竞赛的新型基准数据集，并在基础提示与检索增强生成（RAG）两种模式下评估模型性能。我们的结构化评估框架从功能正确性、语法有效性和执行成功率三个维度，对不同难度等级的挑战任务进行了系统评估。结果表明，在增强版PennyLane数据集支持下，RAG增强模型与标准提示方法在复杂量子算法任务中产生了相近的结果。此外，我们设计了一个多智能体评估流程，通过迭代修正错误解进一步提升了执行成功率。为促进后续研究，我们将公开QHackBench数据集及评估框架与实验结果，以推动人工智能辅助量子编程领域的持续发展。