(Renyi Qu's Master's Thesis) Recent advancements in interpretable models for vision-language tasks have achieved competitive performance; however, their interpretability often suffers due to the reliance on unstructured text outputs from large language models (LLMs). This introduces randomness and compromises both transparency and reliability, which are essential for addressing safety issues in AI systems. We introduce \texttt{Hi-CoDe} (Hierarchical Concept Decomposition), a novel framework designed to enhance model interpretability through structured concept analysis. Our approach consists of two main components: (1) We use GPT-4 to decompose an input image into a structured hierarchy of visual concepts, thereby forming a visual concept tree. (2) We then employ an ensemble of simple linear classifiers that operate on concept-specific features derived from CLIP to perform classification. Our approach not only aligns with the performance of state-of-the-art models but also advances transparency by providing clear insights into the decision-making process and highlighting the importance of various concepts. This allows for a detailed analysis of potential failure modes and improves model compactness, therefore setting a new benchmark in interpretability without compromising the accuracy.

翻译：（任毅的硕士论文）近期，视觉-语言任务中可解释模型的进展已取得具有竞争力的性能；然而，由于依赖大型语言模型（LLMs）生成的非结构化文本输出，其可解释性往往受到影响。这引入了随机性，并损害了透明度和可靠性，而这些特性对于解决AI系统中的安全问题至关重要。我们提出了\texttt{Hi-CoDe}（层次化概念分解），一种旨在通过结构化概念分析增强模型可解释性的新颖框架。我们的方法包含两个主要组成部分：（1）我们使用GPT-4将输入图像分解为结构化的视觉概念层次，从而形成一个视觉概念树。（2）随后，我们采用一组简单的线性分类器，这些分类器基于从CLIP提取的特定概念特征进行分类。我们的方法不仅与最先进模型的性能相当，而且通过提供对决策过程的清晰洞察并突出各类概念的重要性，进一步提升了透明度。这使得对潜在故障模式的详细分析成为可能，并改善了模型的紧凑性，从而在不牺牲准确性的前提下，为可解释性设立了新的基准。