We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully-connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn ``class-specific'' queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INterpretable TRansformer (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via ``multi-head'' cross-attention, INTR could identify different ``attributes'' of a class, making it particularly suitable for fine-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained model are publicly accessible at https://github.com/Imageomics/INTR.

翻译：我们提出了一种Transformer的新型应用范式，使图像分类具有可解释性。与主流分类器依赖最后一个全连接层整合类别信息进行预测不同，我们探索了一种主动方法：让每个类别在图像中自行定位其模式。该思路通过受检测Transformer（DETR）启发的编码器-解码器架构实现。我们学习"类别特异性"查询向量（每类一个）作为解码器输入，使每个类别能通过交叉注意力机制在图像中定位其模式。我们将该方法命名为可解释Transformer（INTR），其实现极为简便且展现出多项引人注目的特性。研究表明，INTR天然促使各类别产生差异化注意力分布；因此交叉注意力权重可提供对预测结果的忠实解释。值得注意的是，通过"多头"交叉注意力机制，INTR能够识别类别的不同"属性"，这使其特别适用于细粒度分类与分析任务——我们在八个数据集上验证了该特性。我们的代码与预训练模型已开源在 https://github.com/Imageomics/INTR。