Recent In-IDE AI coding assistant tools (ACATs) like GitHub Copilot have significantly impacted developers' coding habits. While some studies have examined their effectiveness, there lacks in-depth investigation into the actual assistance process. To bridge this gap, we simulate real development scenarios encompassing three typical types of software development tasks and recruit 27 computer science students to investigate their behavior with three popular ACATs. Our goal is to comprehensively assess ACATs' effectiveness, explore characteristics of recommended code, identify reasons for modifications, and understand users' challenges and expectations. To facilitate the study, we develop an experimental platform that includes a data collection plugin for VSCode IDE and provides functions for screen recording, code evaluation, and automatic generation of personalized interview and survey questions. Through analysis of the collected data, we find that ACATs generally enhance task completion rates, reduce time, improve code quality, and increase self-perceived productivity. However, the improvement is influenced by both the nature of coding tasks and users' experience level. Notably, for experienced participants, the use of ACATs may even increase completion time. We observe that "edited line completion" is the most frequently recommended way, while "comments completion" and "string completion" have the lowest acceptance rates. The primary reasons for modifying recommended code are disparities between output formats and requirements, flawed logic, and inconsistent code styles. In terms of challenges and expectations, optimization of service access and help documentation is also concerned by participants except for functionality and performance. Our study provides valuable insights into the effectiveness and usability of ACATs, informing further improvements in their design and implementation.

翻译：近期，诸如GitHub Copilot等集成开发环境（IDE）内AI编程辅助工具（ACATs）对开发者的编程习惯产生了显著影响。尽管已有研究探讨了其有效性，但对于实际辅助过程的深入调查仍较为匮乏。为弥补这一空白，我们模拟了包含三类典型软件开发任务的真实开发场景，并招募了27名计算机科学专业的学生，以考察他们在三种主流ACATs支持下的行为表现。本研究旨在全面评估ACATs的有效性，探究推荐代码的特征，识别用户修改代码的原因，并理解其面临的挑战与期望。为此，我们开发了一套实验平台，该平台集成了针对VSCode IDE的数据采集插件，并具备屏幕录制、代码评估以及个性化访谈与调查问卷自动生成等功能。通过分析所收集的数据，我们发现ACATs总体上能够提升任务完成率、缩短开发时间、改善代码质量，并增强用户的主观生产力感知。然而，这种提升效果受编码任务性质及用户经验水平的双重影响。值得注意的是，对于经验丰富的参与者，使用ACATs甚至可能延长任务完成时间。我们观察到，“编辑行补全”是最常见的推荐方式，而“注释补全”与“字符串补全”的接受率最低。修改推荐代码的主要原因包括输出格式与需求不匹配、逻辑缺陷以及代码风格不一致。在挑战与期望方面，除功能与性能外，服务访问优化及帮助文档的完善同样受到参与者的关注。本研究为理解ACATs的有效性与可用性提供了重要洞见，并为其设计与实现的进一步改进指明了方向。