Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even more important for both academia and industry to understand how to measure their impact on developer productivity, and to reconsider whether earlier measures and frameworks still apply. This study analyzes the validity of different approaches to evaluating the productivity impacts of AI coding assistants by leveraging mixed-method research. At BNY Mellon, we conduct a survey with 2989 developer responses and 11 in-depth interviews. Our findings demonstrate that a multifaceted approach is needed to measure AI productivity impacts: survey results expose conflicting perspectives on AI tool usefulness, while interviews elicit six distinct factors that capture both short-term and long-term dimensions of productivity. In contrast to prior work, our factors highlight the importance of long-term metrics like technical expertise and ownership of work. We hope this work encourages future research to incorporate a broader range of human-centered factors, and supports industry in adopting more holistic approaches to evaluating developer productivity.

翻译：衡量开发者生产力是一个同时吸引学术研究与工业实践关注的话题。在AI编程助手时代，学术界与工业界理解如何衡量其对开发者生产力的影响，并重新审视早期度量标准与框架是否仍然适用，变得尤为重要。本研究通过混合方法研究，分析了评估AI编程助手生产力影响的不同方法的有效性。在纽约梅隆银行，我们开展了一项涵盖2989份开发者回复的问卷调查和11次深度访谈。我们的研究结果表明，衡量AI生产力影响需要采用多维度方法：调查结果揭示了关于AI工具有效性的矛盾观点，而访谈则归纳出六个不同的因素，这些因素涵盖了生产力的短期与长期维度。与先前工作相比，我们的因素强调了技术专长和工作所有权等长期指标的重要性。我们希望这项工作能鼓励未来研究纳入更广泛的人本因素，并支持工业界采用更全面的方法来评估开发者生产力。