Parameter management is essential for distributed training of large machine learning (ML) tasks. Some ML tasks are hard to distribute because common approaches to parameter management can be highly inefficient. Advanced parameter management approaches -- such as selective replication or dynamic parameter allocation -- can improve efficiency, but to do so, they typically need to be integrated manually into each task's implementation and they require expensive upfront experimentation to tune correctly. In this work, we explore whether these two problems can be avoided. We first propose a novel intent signaling mechanism that integrates naturally into existing ML stacks and provides the parameter manager with crucial information about parameter accesses. We then describe AdaPM, a fully adaptive, zero-tuning parameter manager based on this mechanism. In contrast to prior systems, this approach separates providing information (simple, done by the task) from exploiting it effectively (hard, done automatically by AdaPM). In our experimental evaluation, AdaPM matched or outperformed state-of-the-art parameter managers out of the box, suggesting that automatic parameter management is possible.

翻译：参数管理对于大规模机器学习任务的分布式训练至关重要。部分机器学习任务难以分布式化，因为常见的参数管理方法可能效率极低。先进的参数管理方法（例如选择性复制或动态参数分配）能够提升效率，但通常需要手动集成到每个任务的实现中，并且需要昂贵的先期实验来正确调优。本研究探讨能否避免这两个问题。我们首先提出一种新颖的意图信号机制，该机制能自然融入现有机器学习栈，并为参数管理器提供关于参数访问的关键信息。然后我们描述基于该机制的AdaPM——一种完全自适应、无需调优的参数管理器。与现有系统不同，该方法将信息提供（简单任务，由目标系统完成）与有效利用信息（复杂任务，由AdaPM自动完成）分离开来。实验评估表明，AdaPM能够开箱即用地匹配甚至超越最先进的参数管理器，这证明自动参数管理是可行的。