Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.

翻译：开放词汇模型（如CLIP）通过基于类别（自然语言）名称生成嵌入向量，在零样本分类中展现出强大性能。现有工作主要致力于通过提示工程或引入少量标注下游数据（通过微调）提升模型精度，但鲜有关注类别名称本身的丰富性——当类别标签定义粗糙且信息量不足时可能引发问题。我们提出层级标签集分类方法（简称CHiLS），这是一种专为具有隐式语义层级的数据集设计的零样本分类替代策略。CHiLS包含三个步骤：（i）对于每个类别，利用现有标签层级或通过查询GPT-3生成子类别集合；（ii）将这些子类别视为目标标签，执行标准零样本CLIP流程；（iii）将预测子类别映射回其父类别以生成最终预测。在多个具有层级结构的数据集上，无论是否具备真实层级信息，CHiLS均能提升分类准确率。该方法易于集成到现有CLIP流程中，且无需额外训练成本。代码开源地址：https://github.com/acmi-lab/CHILS。