Many machine learning applications encounter a situation where model providers are required to further refine the previously trained model so as to gratify the specific need of local users. This problem is reduced to the standard model tuning paradigm if the target data is permissibly fed to the model. However, it is rather difficult in a wide range of practical cases where target data is not shared with model providers but commonly some evaluations about the model are accessible. In this paper, we formally set up a challenge named \emph{Earning eXtra PerformancE from restriCTive feEDdbacks} (EXPECTED) to describe this form of model tuning problems. Concretely, EXPECTED admits a model provider to access the operational performance of the candidate model multiple times via feedback from a local user (or a group of users). The goal of the model provider is to eventually deliver a satisfactory model to the local user(s) by utilizing the feedbacks. Unlike existing model tuning methods where the target data is always ready for calculating model gradients, the model providers in EXPECTED only see some feedbacks which could be as simple as scalars, such as inference accuracy or usage rate. To enable tuning in this restrictive circumstance, we propose to characterize the geometry of the model performance with regard to model parameters through exploring the parameters' distribution. In particular, for the deep models whose parameters distribute across multiple layers, a more query-efficient algorithm is further tailor-designed that conducts layerwise tuning with more attention to those layers which pay off better. Extensive experiments on different applications demonstrate that our work forges a sound solution to the EXPECTED problem. Code is available via https://github.com/kylejingli/EXPECTED.

翻译：许多机器学习应用面临这样一种场景：模型提供者需要进一步优化先前训练的模型，以满足本地用户的特定需求。若允许将目标数据输入模型，该问题可简化为标准模型调优范式。然而，在大量实际案例中，目标数据并不与模型提供者共享，但通常可获得关于模型的一些评估反馈，这使问题变得相当困难。本文正式提出一个名为“从限制性反馈中挖掘额外性能”（EXPECTED）的挑战，用以描述此类模型调优问题。具体而言，EXPECTED允许模型提供者通过本地用户（或用户群）的反馈，多次访问候选模型的运行性能。模型提供者的目标是利用这些反馈，最终向本地用户交付一个令人满意的模型。与现有模型调优方法（通常可随时获取目标数据以计算模型梯度）不同，EXPECTED中的模型提供者仅能看到一些反馈，这些反馈可能是简单的标量，例如推理准确率或使用率。为在这种限制性环境下实现调优，我们提出通过探索模型参数的分布来表征模型性能相对于参数的几何特性。特别地，对于参数分布于多个层的深度模型，我们进一步设计了一种更节能的查询算法，该算法逐层进行调优，并更加关注那些回报更高的层。在不同应用上的大量实验表明，我们的工作为EXPECTED问题提供了可靠的解决方案。代码可通过https://github.com/kylejingli/EXPECTED获取。