We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks), and code-only $\approx$ documented on code-derivable tasks, validating the design. The framework is open-source and applicable to any well-documented Python repository.

翻译：我们提出Code-QA-Bench，一个全自动化的代码理解基准合成框架，用于在仓库级代码理解中分离真正的代码理解与文档回忆及预训练记忆。该框架有两项方法论贡献：（1）答案优先生成管道——配备工具的智能体先探索源代码生成经过验证的黄金答案，再推导问题，确保每项任务都基于真实代码结构；（2）三条件实验设计——在闭卷（无仓库）、仅代码（删除文档）和含文档（完整仓库）三种条件下评估智能体，通过差值直接量化文档效用与记忆效应。我们从SWE-Bench的10个Python仓库中生成528项代码可推导任务和100项文档依赖任务，由LLM裁判基于准确性、完整性和特异性进行评分。对四个前沿模型的实验表明：代码访问是主导因素（比闭卷平均提升+0.23），文档提供适度额外收益（在文档依赖任务上+0.071），且在代码可推导任务上仅代码≈含文档，验证了设计有效性。该框架为开源实现，适用于任何文档完善的Python仓库。