With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer's toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information. In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application's issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.

翻译：凭借其卓越的自然语言处理能力，基于大型语言模型（LLM）的工具（如ChatGPT和Co-Pilot）已迅速成为软件开发人员工具包中不可或缺的资源。尽管近期研究表明这些工具能释放潜在的生产力提升，用户仍会遭遇诸如泛化或错误答案等缺陷。此外，为追求更优响应而进行的提示工程工作往往耗时巨大，挤占了编写真正价值代码的宝贵时间。为应对这些挑战，一种基于LLM的新型工具正在兴起。这类工具通过微调或利用情境化信息丰富用户提示等技术来缓解上述缺陷。本文深入探讨了一个软件开发生团队在创建名为CodeBuddy的基于检索技术的情境化LLM应用过程中积累的经验教训。尽管团队缺乏LLM相关应用的专业经验，但在四个月内从零构建了该产品。初始版本发布后，我们与负责代码生成组件的开发团队进行了交流。通过访谈和应用问题追踪系统分析，我们揭示了LLM应用开发团队可能面临的诸多有趣挑战。例如，我们发现了三类主要经验：基于LLM的经验、用户相关经验和技术经验。理解这些经验教训，将有助于软件开发团队更从容地构建基于LLM的应用。