With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer's toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information. In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application's issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.

翻译：凭借其卓越的自然语言处理能力，基于大型语言模型（LLM）的工具（如ChatGPT和Co-Pilot）迅速成为软件开发人员工具包中不可或缺的资源。尽管近期研究表明这些工具能释放潜在的生产力提升，但用户仍会遇到一些问题，例如通用性或错误的回答。此外，为追求更佳回答，用户往往需投入大量精力进行提示工程，从而占用了本应用于编写带来实际价值代码的时间。为解决这些挑战，一类新兴工具正应运而生，它们基于LLM构建，通过微调或用情境化信息丰富用户提示等技术来缓解上述缺陷。本文深入探讨了一个软件开发团队在尝试创建此类基于LLM、采用检索技术的情境化应用（名为CodeBuddy）过程中所汲取的经验教训。该团队在为期四个月的时间里，尽管先前缺乏开发基于LLM应用的专业经验，却从零开始构建了该产品。在初始版本发布后，我们与负责代码生成组件的开发团队进行了交流。通过访谈及对应用问题追踪系统的分析，我们揭示了开发基于LLM应用的团队可能遇到的多项有趣挑战。例如，我们发现了三类主要经验教训：基于LLM的教训、基于用户的教训以及技术层面的教训。理解这些经验教训，可使软件开发团队在构建基于LLM的应用时准备得更加充分。