The simplest and often most effective way of parallelizing the training of complex machine learning models is to execute several training instances on multiple machines, possibly scanning the hyperparameter space to optimize the underlying statistical model and the learning procedure. Often, such a meta learning procedure is limited by the ability of accessing securely a common database organizing the knowledge of the previous and ongoing trials. Exploiting opportunistic GPUs provided in different environments represents a further challenge when designing such optimization campaigns. In this contribution we discuss how a set of RestAPIs can be used to access a dedicated service based on INFN Cloud to monitor and possibly coordinate multiple training instances, with gradient-less optimization techniques, via simple HTTP requests. The service, named Hopaas (Hyperparameter OPtimization As A Service), is made of web interface and sets of APIs implemented with a FastAPI back-end running through Uvicorn and NGINX in a virtual instance of INFN Cloud. The optimization algorithms are currently based on Bayesian techniques as provided by Optuna. A Python front-end is also made available for quick prototyping. We present applications to hyperparameter optimization campaigns performed combining private, INFN Cloud and CINECA resources.

翻译：在复杂机器学习模型的训练中，并行化的最简单且通常最有效的方法是在多台机器上执行多个训练实例，通过扫描超参数空间来优化底层统计模型和学习过程。这种元学习过程通常受限于安全访问一个组织先前和进行中试验知识的公共数据库的能力。在设计此类优化方案时，利用不同环境下提供的临时GPU构成了一项额外挑战。本文讨论如何通过一组RestAPI访问基于INFN云构建的专用服务，借助无梯度优化技术，通过简单的HTTP请求监控并可能协调多个训练实例。该服务名为Hopaas（超参数优化即服务），由Web界面和API集合组成，后端基于FastAPI实现，通过Uvicorn和NGINX在INFN云的虚拟实例中运行。优化算法目前基于Optuna提供的贝叶斯技术。同时提供Python前端以便快速原型开发。我们展示了结合私有资源、INFN云和CINECA资源进行的超参数优化应用实例。