Deploying machine learning (ML) on diverse computing platforms is crucial to accelerate and broaden their applications. However, it presents significant software engineering challenges due to the fast evolution of models, especially the recent Large Language Models (LLMs), and the emergence of new computing platforms. Current ML frameworks are primarily engineered for CPU and CUDA platforms, leaving a big gap in enabling emerging ones like Metal, Vulkan, and WebGPU. While a traditional bottom-up development pipeline fails to close the gap timely, we introduce TapML, a top-down approach and tooling designed to streamline the deployment of ML systems on diverse platforms, optimized for developer productivity. Unlike traditional bottom-up methods, which involve extensive manual testing and debugging, TapML automates unit testing through test carving and adopts a migration-based strategy for gradually offloading model computations from mature source platforms to emerging target platforms. By leveraging realistic inputs and remote connections for gradual target offloading, TapML accelerates the validation and minimizes debugging scopes, significantly optimizing development efforts. TapML was developed and applied through a year-long, real-world effort that successfully deployed significant emerging models and platforms. Through serious deployments of 82 emerging models in 17 distinct architectures across 5 emerging platforms, we showcase the effectiveness of TapML in enhancing developer productivity while ensuring model reliability and efficiency. Furthermore, we summarize comprehensive case studies from our real-world development, offering best practices for developing emerging ML systems.

翻译：在多样化计算平台上部署机器学习（ML）对于加速和拓展其应用至关重要。然而，由于模型（尤其是近期的大语言模型，LLMs）的快速演进以及新兴计算平台的出现，这带来了显著的软件工程挑战。当前的ML框架主要面向CPU和CUDA平台设计，在支持Metal、Vulkan和WebGPU等新兴平台方面存在巨大缺口。传统自下而上的开发流程难以弥补这一时效性差距，为此我们提出了TapML——一种旨在简化ML系统在多样化平台上部署的自上而下的方法与工具集，专注于提升开发者生产力。与涉及大量手动测试和调试的传统自下而上方法不同，TapML通过测试雕刻自动进行单元测试，并采用基于迁移的策略，逐步将模型计算从成熟的源平台迁移到新兴的目标平台。通过利用真实输入和远程连接实现渐进式目标迁移，TapML加速了验证过程并缩小了调试范围，显著优化了开发工作。经过历时一年的实际开发与应用，TapML成功部署了重要的新兴模型与平台。通过在5个新兴平台上部署涵盖17种不同架构的82个新兴模型，我们展示了TapML在提升开发者生产力的同时确保模型可靠性与效率方面的有效性。此外，我们总结了实际开发中的综合案例研究，为新兴ML系统的开发提供了最佳实践。