Programming assistants powered by large language models have improved dramatically, yet existing benchmarks still evaluate them in narrow code-generation settings. Recent efforts such as InfiBench and StackEval rely on Stack Overflow questions and remain limited to single-turn interactions, manually curated data, and isolated snippets rather than full project environments. We introduce CodeAssistBench (CAB), the first benchmark for evaluating multi-turn, project-grounded programming assistance at scale. CAB automatically constructs datasets from GitHub issues tagged as questions, using an LLM-driven pipeline that filters noise, extracts runnable contexts, builds executable containers, and verifies environment correctness. This enables continuous, automated expansion across diverse repositories without manual intervention. Using CAB, we create a testbed of 3,286 real-world issues across 214 repositories, spanning seven languages. Evaluating state-of-the-art models reveals a substantial gap: while models achieve 70-83% accuracy on Stack Overflow-style questions, they solve only 7.22-16.49% of CAB issues from post-training-cutoff repositories. These results highlight a fundamental challenge: current LLMs struggle to provide assistance in realistic, project-specific contexts despite strong performance on traditional Q&A benchmarks. CAB provides a scalable, reproducible framework for advancing research in multi-turn, codebase-grounded programming agents. The benchmark and pipeline are fully automated and publicly available at https://github.com/amazon-science/CodeAssistBench/.

翻译：基于大语言模型的编程助手已取得显著进步，但现有基准测试仍局限于狭窄的代码生成场景进行评估。近期研究如InfiBench和StackEval依赖于Stack Overflow问题，且仅限于单轮交互、人工筛选数据以及独立代码片段而非完整项目环境。本文提出CodeAssistBench (CAB)——首个用于大规模评估基于项目的多轮编程辅助的基准测试。CAB通过LLM驱动的流程自动从标记为问题的GitHub issues构建数据集，该流程可过滤噪声、提取可运行上下文、构建可执行容器并验证环境正确性。这使得无需人工干预即可在不同代码库中实现持续、自动化的扩展。基于CAB，我们创建了涵盖214个代码库、涉及七种编程语言的3,286个真实issues测试集。对前沿模型的评估揭示了显著差距：尽管模型在Stack Overflow式问题上达到70-83%的准确率，但在训练截止日期后的代码库中，仅能解决7.22-16.49%的CAB问题。这些结果表明了根本性挑战：尽管在传统问答基准上表现优异，当前LLM在真实项目特定场景中提供辅助的能力仍存在不足。CAB为推进基于代码库的多轮编程智能体研究提供了可扩展、可复现的框架。该基准测试与流程完全自动化，公开发布于https://github.com/amazon-science/CodeAssistBench/。