Large Language Models (LLMs) are increasingly embedded into software products across diverse industries, enhancing user experiences, but at the same time introducing numerous challenges for developers. Unique characteristics of LLMs force developers, who are accustomed to traditional software development and evaluation, out of their comfort zones as the LLM components shatter standard assumptions about software systems. This study explores the emerging solutions that software developers are adopting to navigate the encountered challenges. Leveraging a mixed-method research, including 26 interviews and a survey with 332 responses, the study identifies 19 emerging solutions regarding quality assurance that practitioners across several product teams at Microsoft are exploring. The findings provide valuable insights that can guide the development and evaluation of LLM-based products more broadly in the face of these challenges.

翻译：大语言模型正日益嵌入到各行各业的软件产品中，以提升用户体验，但同时也为开发者带来了诸多挑战。LLM 的独特特性迫使习惯于传统软件开发和评估的开发者走出舒适区，因为 LLM 组件打破了关于软件系统的标准假设。本研究探讨了软件开发者正在采用以应对所遇挑战的新兴解决方案。通过采用混合研究方法，包括 26 次访谈和一项收到 332 份回复的问卷调查，本研究识别了微软多个产品团队的实践者正在探索的、关于质量保证的 19 项新兴解决方案。这些发现为更广泛地指导基于 LLM 的产品在面对这些挑战时的开发和评估提供了宝贵的见解。