This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and WizardCoder (open-source, 15B parameters) models on the automated accomplishment of various MLOps functionalities in different settings. We perform a benchmarking study that assesses the ability of these models to: (1) adapt existing code samples (Inlining) with component-specific MLOps functionality such as MLflow and Weights & Biases for experiment tracking, Optuna for hyperparameter optimization etc., and (2) perform the task of Translation from one component of an MLOps functionality to another, e.g., translating existing GitPython library based version control code to Data Version Control library based. We also propose three different approaches that involve teaching LLMs to comprehend the API documentation of the components as a reference while accomplishing the Translation tasks. In our evaluations, the gpt-3.5-turbo model significantly outperforms WizardCoder by achieving impressive Pass@3 accuracy in model optimization (55% compared to 0% by WizardCoder), experiment tracking (100%, compared to 62.5% by WizardCoder), model registration (92% compared to 42% by WizardCoder) and hyperparameter optimization (83% compared to 58% by WizardCoder) on average, in their best possible settings, showcasing its superior code adaptability performance in complex MLOps tasks.

翻译：本文探索了当前大语言模型将机器学习运维（MLOps）功能集成到机器学习训练代码库中的可能性。我们评估了OpenAI（gpt-3.5-turbo）和WizardCoder（开源模型，150亿参数）在不同设置下自动实现多种MLOps功能的表现。通过基准测试研究，我们评估了这些模型的能力：（1）将组件特定的MLOps功能（如用于实验追踪的MLflow与Weights & Biases、用于超参数优化的Optuna等）适配到现有代码样本（内联模式）中；（2）执行MLOps功能组件间的转换任务，例如将基于GitPython库的版本控制代码转换为基于Data Version Control库的代码。我们还提出了三种不同方法，通过教导LLMs理解组件API文档作为参考来完成转换任务。在评估中，gpt-3.5-turbo模型显著优于WizardCoder，在最佳设置下，其在模型优化（55%对比WizardCoder的0%）、实验追踪（100%对比62.5%）、模型注册（92%对比42%）和超参数优化（83%对比58%）任务中的平均Pass@3准确率均表现突出，展现了其在复杂MLOps任务中卓越的代码自适应性能。