Adapting models deployed to test distributions can mitigate the performance degradation caused by distribution shifts. However, privacy concerns may render model parameters inaccessible. One promising approach involves utilizing zeroth-order optimization (ZOO) to train a data adaptor to adapt the test data to fit the deployed models. Nevertheless, the data adaptor trained with ZOO typically brings restricted improvements due to the potential corruption of data features caused by the data adaptor. To address this issue, we revisit ZOO in the context of test-time data adaptation. We find that the issue directly stems from the unreliable estimation of the gradients used to optimize the data adaptor, which is inherently due to the unreliable nature of the pseudo-labels assigned to the test data. Based on this observation, we propose pseudo-label-robust data adaptation (SODA) to improve the performance of data adaptation. Specifically, SODA leverages high-confidence predicted labels as reliable labels to optimize the data adaptor with ZOO for label prediction. For data with low-confidence predictions, SODA encourages the adaptor to preserve data information to mitigate data corruption. Empirical results indicate that SODA can significantly enhance the performance of deployed models in the presence of distribution shifts without requiring access to model parameters.

翻译：将模型部署到测试分布中进行适配可以缓解由分布偏移导致的性能下降。然而，隐私问题可能使得模型参数不可访问。一种有前景的方法是采用零阶优化（ZOO）训练数据适配器，以调整测试数据使其适配已部署的模型。然而，由于数据适配器可能导致数据特征受损，基于ZOO训练的数据适配器通常只能带来有限的改进。为解决这一问题，我们重新审视了ZOO在测试时数据适配中的运用。我们发现该问题直接源于用于优化数据适配器的梯度估计不可靠，而这本质上是由测试数据伪标签的不可靠性导致的。基于此观察，我们提出伪标签鲁棒数据适配（SODA）以提升数据适配性能。具体而言，SODA利用高置信度预测标签作为可靠标签，通过ZOO优化数据适配器以进行标签预测；对于低置信度预测的数据，SODA则促使适配器保留数据信息以减轻数据损伤。实验结果表明，SODA能在无需访问模型参数的情况下，显著提升部署模型在分布偏移下的性能。