Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.

翻译：开放词汇模型（如CLIP）通过根据类别（自然语言）名称生成嵌入，在零样本分类中展现出强大性能。现有研究主要集中于通过提示工程或结合少量标注的下游数据（微调）提升模型精度，但鲜有关注类名本身的丰富性——当类别标签定义粗糙且缺乏信息量时，这一问题尤为突出。我们提出层级标签集分类（简称CHiLS），这是一种专为隐式语义层级结构数据集设计的零样本分类替代策略。CHiLS包含三个步骤：（i）利用已有标签层级或通过查询GPT-3为每个类别生成子类集合；（ii）执行标准零样本CLIP流程，将子类视为目标标签；（iii）将预测子类映射回其父类以输出最终结果。在多个具有层级结构的数据集上，无论是否依赖真实层级信息，CHiLS均能提升分类准确率。该方法可轻松集成至现有CLIP流程且无需额外训练成本。代码开源地址：https://github.com/acmi-lab/CHILS。