Deploying machine learning (ML) on diverse computing platforms is crucial to accelerate and broaden their applications. However, it presents significant software engineering challenges due to the fast evolution of models, especially the recent \llmfull{s} (\llm{s}), and the emergence of new computing platforms. Current ML frameworks are primarily engineered for CPU and CUDA platforms, leaving a big gap in enabling emerging ones like Metal, Vulkan, and WebGPU. While a traditional bottom-up development pipeline fails to close the gap timely, we introduce TapML, a top-down approach and tooling designed to streamline the deployment of ML systems on diverse platforms, optimized for developer productivity. Unlike traditional bottom-up methods, which involve extensive manual testing and debugging, TapML automates unit testing through test carving and adopts a migration-based strategy for gradually offloading model computations from mature source platforms to emerging target platforms. By leveraging realistic inputs and remote connections for gradual target offloading, TapML accelerates the validation and minimizes debugging scopes, significantly optimizing development efforts. TapML was developed and applied through a year-long, real-world effort that successfully deployed significant emerging models and platforms. Through serious deployments of 82 emerging models in 17 distinct architectures across 5 emerging platforms, we showcase the effectiveness of TapML in enhancing developer productivity while ensuring model reliability and efficiency. Furthermore, we summarize comprehensive case studies from our real-world development, offering best practices for developing emerging ML systems.

翻译：将机器学习（ML）部署到多样化的计算平台上，对于加速和拓展其应用至关重要。然而，由于模型（尤其是近期的大语言模型（LLM））的快速演进以及新计算平台的出现，这带来了巨大的软件工程挑战。当前的ML框架主要针对CPU和CUDA平台设计，在支持Metal、Vulkan和WebGPU等新兴平台方面存在巨大空白。传统的自下而上开发流程无法及时填补这一空白，我们引入了TapML，这是一种自上而下的方法和工具集，旨在简化ML系统在多样化平台上的部署，并优化开发者生产力。与传统的自下而上方法（涉及大量手动测试和调试）不同，TapML通过测试雕琢自动化单元测试，并采用基于迁移的策略，将模型计算从成熟的源平台逐步迁移到新兴的目标平台。通过利用真实输入和远程连接实现逐步的目标卸载，TapML加速了验证过程并最小化了调试范围，显著优化了开发工作。TapML经过长达一年的实际开发与应用，成功部署了重要的新兴模型和平台。通过在5个新兴平台上实际部署了横跨17种不同架构的82种新兴模型，我们展示了TapML在提升开发者生产力同时确保模型可靠性和效率方面的有效性。此外，我们总结了实际开发中的综合案例研究，为开发新兴ML系统提供了最佳实践。