[Context] Systems incorporating Machine Learning (ML) models, often called ML-enabled systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited, especially for activities surrounding ML model dissemination. [Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring of ML life cycle phases. [Method] We conducted an international survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered a total of 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We analyzed contemporary practices using bootstrapping with confidence intervals and conducted qualitative analyses on the reported problems applying open and axial coding procedures. [Results] Practitioners perceive the model deployment and monitoring phases as relevant and difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and the selection of suitable metrics. [Conclusion] Our results help provide a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner.
翻译:[背景] 集成机器学习模型的系统(通常称为ML-Enabled系统)已变得十分普遍。然而,关于ML-Enabled系统在实际工程中如何构建的经验证据仍然有限,特别是围绕机器学习模型部署阶段的活动。[目标] 我们调查与ML模型部署相关的当代工业实践和问题,重点关注ML生命周期的模型部署与监控阶段。[方法] 我们开展了一项国际调查,以收集从业者对ML-Enabled系统工程实践的见解。共收集来自25个国家的188份完整回复。我们分析了模型部署与监控阶段的现状及报告的问题。通过使用置信区间自助法分析当代实践,并应用开放编码和主轴编码程序对报告的问题进行定性分析。[结果] 从业者认为模型部署与监控阶段既重要又困难。在模型部署方面,模型通常作为独立服务部署,MLOps原则的采用有限。报告的问题包括为生产部署设计基础设施架构以及集成遗留应用的困难。在模型监控方面,许多生产环境中的模型并未被监控。主要监控的方面包括输入、输出和决策。报告的问题涉及监控实践的缺失、需要创建自定义监控工具以及选择合适的指标。[结论] 我们的结果有助于更好地理解实际实践与问题,并以问题驱动的方式支持指导ML部署与监控研究。