Zero-shot inference is a powerful paradigm that enables the use of large pretrained models for downstream classification tasks without further training. However, these models are vulnerable to inherited biases that can impact their performance. The traditional solution is fine-tuning, but this undermines the key advantage of pretrained models, which is their ability to be used out-of-the-box. We propose RoboShot, a method that improves the robustness of pretrained model embeddings in a fully zero-shot fashion. First, we use language models (LMs) to obtain useful insights from task descriptions. These insights are embedded and used to remove harmful and boost useful components in embeddings -- without any supervision. Theoretically, we provide a simple and tractable model for biases in zero-shot embeddings and give a result characterizing under what conditions our approach can boost performance. Empirically, we evaluate RoboShot on nine image and NLP classification tasks and show an average improvement of 15.98% on worst group accuracy, with trivial decrease in overall accuracy over several zero-shot baselines. Additionally, we demonstrate that RoboShot is compatible with a variety of pretrained and language models and propose a way to further boost performance with a zero-shot adaptation variant.

翻译：零样本推理是一种强大的范式，它允许利用大型预训练模型直接执行下游分类任务而无需额外训练。然而，这些模型容易受到继承偏差的影响，进而损害其性能。传统解决方案是微调，但这会削弱预训练模型的核心优势——即开箱即用的能力。我们提出RoboShot方法，该方法以完全零样本的方式提升预训练模型嵌入的鲁棒性。首先，我们使用语言模型从任务描述中获取有用的见解；这些见解被嵌入后，用于移除嵌入中有害成分并增强有益成分——整个过程无需任何监督。理论上，我们为零样本嵌入中的偏差提供了一个简单且可处理的模型，并给出了描述该方法在何种条件下能够提升性能的特征结果。在实证方面，我们在九个图像和NLP分类任务上评估了RoboShot，结果显示在最差组准确率上平均提升15.98%，同时在多个零样本基线中总体准确率几乎无下降。此外，我们证明了RoboShot与多种预训练模型和语言模型兼容，并提出了通过零样本自适应变体进一步提升性能的方法。